if we specify start_time as well as conditions in autosys job, which get preference over another ??
For e.g. lets say start time for my job A is 06:00 AM but this job is also dependent on another job B (via conditions param) . But Job B gets becomes successful at 06:30 AM.
So my job will start running at 06:00 AM or 06:30 AM ?
My guess is start time as well as conditions, both will be considered while running the job. So job should start at 06:30 AM. But i still want to be sure.
-Gaurav
Start_time:- Job started means Activated or might be run, (if conditions are already met) at starting time.
Conditions:- Means job would run after given condition mate.
Example:-
Suppose job A starting time is 11:00 PM and its condition is s(B)---->B should success, B is another job that would complete at 11:30 PM. In this scenario A started at 11:00 PM but it would in active state for 30 min. than it would in running state.
Your job will be in Activated state at 6:00 AM and will wait until B completes...but
consider a situation when you have another job C which is running before B. In case if C is running long and B is delayed and is not in Activated state (it is in sucessfull since run from, say, yesterday and waiting for C), you job A will also trigger as it's already 6 and B is in success (even from yesterday).
In order to avoid this, you have two ways:
1) Make B activated before A
2) Make a condition s(A, )
Related
I am quite confused about time concept in airflow. As far as I know, there are at least 5 different time concept in airflow:
UTC time in web UI
Execution time, e.g. start_date, in DAG file
RUN time.
time saved in database
system time
Could anyone explain which are the same and which are different?
The UTC time in the Web UI(1) is the system time (5) being presented.
The run time (3) shows when a task was actually run in UTC and this is the time saved in the database (4) as it corresponds to when a certain task was actually ran in a DAG.
The start_date should be a static non-changing date as it corresponds to when the DAG was first started and should be backfilled if certain parameters like catchup or depends_on_past is set to true.
So 5 and 1 are the same, 3 and 4 are the same, and 5 is it's own entity of time.
Predecessor job(JOB_A) running M-F
JOB_B is set up to run M-TH after JOB_A completion
We have job(JOB_C) that needs to run only on FRIDAY's after JOB_A completion.
Because of FRIDAY only we have to use DATE condition with DAY and TIME.
JOB_C is triggering at the Time and not waiting for the JOB_A completion.
(JOB_A dependent on another job so it may run any time between 19:00 to 23:00
job run time may be 5Min only)
Can Autosys handle this?
Regards
Please correct me if I misunderstood your problem.
From my understanding you want JOB_C to run on Friday, only after the completion of JOB_A in that case you can add the following attributes to the JIL of JOB_C
days_of_week: fr
conditions: s(JOB_A)
you can use conditions : d(JOB_A) if there is no actual dependency on JOB_A other than the order in which they run.
d(JOB_A) is for DONE, that will satisfy the condition as soon as JOB_A had completed irrespective of the exit status/code
s(JOB_A) is for SUCCESS, so it will only satisfy if JOB_A had completed successfully
You might want to have a dummy job, let's say JOB_D, just to indicate Friday. The command does not matter, what matter is the schedule - FRIDAY, 00:00. Add this job as a second predecessor for JOB_C with condition SUCCESS IN LAST 24 HOURS - like
job: JOB_C
condition: d(JOB_A) & s(JOB_D, 24.00)
...
Job B
Condition s(Job A, 9.00)
Start time: 9:15
Job A has run within 9 hours. Sometimes the Job A starts running again just before the scheduled start time of Job B (9:15). Because of this, Job B cannot start as Job A is RU instead of SU.
What kind of condition on Job B would solve this issue?
Is your jobB should be really dependent with the success of jobA? if yes, then all you need to do is to wait for the jobA to be completed, and jobB will automatically start after that. If no, you can simply remove the jobA in the run condition of jobB and remain the start time of jobB as 9:15. Also, are your jobs reside in the same box? Is your jobA has multiple run time in a day?
I have job A that have time condition as 6:00 and predecessor as job B(this completes mostly by 4:00). Yesterday job B did not ran and today after job B completion job A started as time condition was satisfied on previous day itself. I want to know whether is there any way to make this time condition to be unsatisfied after that particular day completion so that there wont be any issues in next day?
If you are using R11 version, then you can use lookback condition(refer CA manual for details)
syntax is
status(job_name, hhhh.mm)
eg
condition: success(job_b,12.00)
If job_b has become success in last 12 hrs, then condition is satisfied, else its not and the job_a doesnt run.
Hope this will do.
I have an Hive script that I am executing using Oozie coordinator every 10 minutes. When I launched my Oozie coordinator-app, suppose I have started at 08:03, the first workflow starts at that time, the next 08:13, and then 08:23, and so on.
What I want is to execute the workflow every clock time hh:mm, where mm is divisible by 10. Assuming the same scenario above, what I want to happen is this: the first workflow will execute at 08:10, and then 08:20, and so on.
How do I do this in Oozie? How about when every 5 minutes (The last m of the minute is either 5 or 0)? Thanks for your input.
In order to run a coordinator job at a frequency, you can use the following directives
<coordinator-app name="app" frequency="10" start="2015-07-10T12:00Z" end="2016-01-01T00:00Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
This would run every 10 minutes, starting exactly at 12:00 UTC time today. The same goes for running every 5 minutes, just replace frequency="10" with frequency="5". To to have it run every Nth minute divisible by M, you will have to ensure that your start parameter is set correctly.
Another option if you are using a more recent version of Oozie (4.1.0) would be to use the cron like scheduler. This would allow you to schedule Oozie coordinators in a cron-like fashion if you're familiar. See http://blog.cloudera.com/blog/2014/04/how-to-use-cron-like-scheduling-in-apache-oozie/ and https://issues.apache.org/jira/browse/OOZIE-1306