I have given the below conditions to run every 4 hours during those days, but the problem is the job is running at 01:00, 05:00 and 08:58 on Wednesdays. I expect the job to run from 09:00 AM Wednesday till 08:59 AM Thursday.
insert_job: file_watcher.0000.000 job_type: BOX
date_conditions: 1
days_of_week: we,th
start_times: "09:00,13:00,17:00,21:00,01:00,05:00,08:59"
run_window: "09:00 - 08:59"
description: "File watcher job"
Could someone guide me how to modify the conditions to achieve this.
This is not possible by 1 job definition.
Suggest you, create 2 jobs one for Wed and Thurs with their respective timings.
Related
I have one problem, I need to launch one DAG every first day of month, but I have one problem, the DAG started on 1 October but executed that day on 1 November, I need that 1 October execute 1 October and 1 November execute 1 November, and not delay the execution one month.
My scheduler was: '0 10 1 * *'
Thanks
This is how Airflow works.
Airflow schedule DAGs at the end of the interval.
So if you have:
DAG(
dag_id='tutorial',
schedule_interval='0 10 1 * *',
start_date=datetime(2021, 10, 1),
)
The first run will start on 2021-11-01 - this run will have execution date of 2021-10-01. This behavior is consistent with how data pipelines work. In November you will want to process October data. Or in the terminology that I mentioned before - Your monthly interval starts on beginning of October it ends in November so at the beginning of November you can run the job that process October data.
That said - In the job itself you can process any interval you wish. For that you can use Airflow macros.
In simple words if you want your first DAG run to start on 2021-10-01 you should set start_date=datetime(2021, 9, 1)
Starting from Airflow 2.2.0 there was enhancement in that area.
Airflow decoupled the "When to run" from the "What interval to process" with the completion of AIP- 39 Richer Scheduler. You can read about the concept of Timetables in this doc.
I am having issues with a DAG created, I am trying to create a DAG to run weekly on Mondays.
This is the current setting of the DAG:
dag = DAG(
"NAME_OF_THE_DAG",
description="description",
default_args=default_args,
start_date="Fri, 01 May 2020 00:00:00 GMT",
end_date=None,
schedule_interval="30 3 * * 1",
catchup=True,
)
The DAG runs the first task on the first Monday of the May (4th of May 2020), but then is not executing the following week.
Does anyone what could be? I thought it was something with qeued tasks, but after cleaning the tasks, the scheduled didn't triggered the expected next execution.
Airflow schedules each task at the end of each interval. So your runs will be...
Monday, May 3 # 3:30 will execute on Monday, May 10 # 3:30
Monday, May 10 # 3:30 will execute on Monday, May 17 # 3:30
Monday, May 17 # 3:30 will execute on Monday, May 24 # 3:30
etc.
Why is this? Since Airflow is a tool built for data workflows where each task operates on a slice of time, and the data for a slice of time is not ready until that slice of time has ended. So for instance with your schedule, you define a window for May 3 to May 10, and that data won't be ready until May 10.
I've found pages 11-14 of this PDF to do the best job of explaining: https://drive.google.com/file/d/1DVN4HXtOC-HXvv00sEkoB90mxLDnCIKc/view
I had configured my cronjob to run in every first Monday of the month 8:40am as below
40 08 1-7 * 1 /fs/test/testtime.sh
But it not only run on Monday, it also run on today which is Tuesday.
Is there anything i miss out?
From the man page for crontab (my emphasis):
Note: The day of a command's execution can be specified by two fields - day of month, and day of week. If both fields are restricted (i.e., aren't *), the command will be run when either field matches the current time.
For example, 30 4 1,15 * 5 would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.
So, in your case, the job runs on every one of the first seven days in each month, plus every Monday.
You can do what you wish by adding an AND condition in the command rather than relying on an OR condition in the time specification, something like:
40 08 1-7 * * test $(date +\%u) -eq 1 && /fs/test/testtime.sh
This will run the actual cron job on all those days (first seven days in each month) but the payload (the script) will only run if the day is Monday.
I need to create an autosys job that run on Saturday and Sunday only at Noon and 11:00 p.m. CST
below is the script that I wrote based on what I saw online.
I'm new to this , can someone else please check and confirm if there's any other thing that I'm missing that can cause some issue?
insert_job: name job_type: CMD
command: /directory/zzz/xxx.ksh /directory/xxx/xxx
machine: ZZZ
owner: user
permission:
date_conditions: 1
days_of_week: sa,su
start_mins: 5,20,35,50
run_window: "12:00 , 23:00"
condition: s(XXX_XXX_CONTROL)
description: "pull xxx files"
std_out_file: $dummy.out
std_err_file: $dummy.err
alarm_if_fail: 1
profile: /directory/directory.profile
Your block for determining which days your jobs will run looks correct:
date_conditions: 1
days_of_week: sa,su
But your block for determining when on those days your job runs is off:
start_mins: 5,20,35,50 # specifies how many minutes after the top of an hour a job starts
run_window: "12:00 , 23:00"
As your JIL is now, the job will run 4 times an hour for 11 hours:
12:05, 12:20, 12:35, 12:50, 13:05, 13:20, ..., 22:35, and 22:50
You most likely want specific start_times:
start_times: "12:00, 23:00"
start_times and start_mins can't be defined in the same JIL, and run_window would be extraneous here, so it can be omitted along with start_mins.
You can also take a look at the must_start_times attribute if you want an alarm if your jobs didn't start on time.
I am doing some work on a fairly old system and need to know when a couple of processes started. When I use "ps -ef" one says October 18 and the other March 23. We haven't got to October 18 this year so I'm not sure if this is October 18 last year or the previous year. The uptime command is showing 2419 days (6.6 years!) so it's possible the Oct 18 is from earlier than 2012. This is a HP-UX system. I have done a bit of googling and none of the answers I came across worked, eg ps -o, looking in the /proc dir.
cfgmgr 9947 9943 3 Mar 23 ? 6831:32 /home/cfgmgr/bin/snmpagt
root 24338 1 0 Oct 18 ? 2628:13 /usr/sbin/snmpdm -tcplocal
PS can only show the date because as per PS documentation at MAN page
'Only the year will be displayed if the process was not started the same year ps was invoked, or "mmmdd" if it was not started the same day, or "HH:MM" otherwise.'