I have an Airflow dag that I need to schedule at the same time whether it is standard or daylight savings time, but the airflow server and cron scheduling is based on utc. It needs to run after a database restore occurs and that server runs the restore based on the local time.
This is causing me to have to schedule it an hour later during daylight savings time. Is there a way to have airflow run the job at the same hour whether it is daylight savings time or standard time?
For example, the database restore completes at noon and 8PM daily. This time is the same whether it is PDT or PST. Any way to do this in cron so that the time the job runs is always at noon and 8pm?
You can set the timezone in Airflow configuration, in airflow.cfg, to any IANA timezone. The default is utc.
The UI instead only runs in UTC.
More info on timezones in Airflow at https://airflow.apache.org/timezone.html
Related
I have a airflow dag-1 that runs approximately for week and dag-2 that runs every day for few hours. When the dag-1 is running i cannot have the dag-2 running due to API limit rate (also dag-2 is supposed to run once dag-1 is finished).
Suppose the dag-1 is running already, then dag-2 that is supposed to run everyday fails, is there a way i can schedule the dag dependencies in a right way?
Is it possible to stop dag-1 temporarily(while running) when dag-2 is supposed to start and then run dag-1 again without manual interruption?
One of the best way is to use the defined pool ..
Lets say if you have a pool named: "specefic_pool" and allocate only one slot for it.
Specify the pool name in your dag bash command (instead of default pool, please use newly created pool) By that way you may over come of running both the dags parallel .
This helps whenever Dag1 is running Dag2 will never be triggered until pool is free or if the dag2 picked the pool until dag2 is completed dag1 is not going to get triggered.
I'm using airflow v1.9.0.
When I trigger a dag from the web-ui, the execution date is in my local time, which means that for airflow the task is scheduled in the future.
How can I configure the webserver to use it's local time over mine?
Do Unix timestamps share the same value in different zones.
Basically if a computer is running in Japan under a Japanese timezone and a computer is running in UK is the unix timestamp the same for them for the particular instance in time. I believe it is as UNIX time stamp defaults to UTC.
In other words in my mobile application is it safe to calculate time differences on the client side based on the unix time stamp? It wouldn't be if the current unix time stamp was different depending on the timezone.
Yes, assuming that two servers have their time synchronized using some NTP service.
NOTE:
In mysql, unix_timestamp() execution is free of local time_zone and returns result w.r.t UTC but unix_timestamp(DATE) function assumes that specified DATE is in local time_zone and thus first converts it to UTC and then returns the result.
We have multiple jobs configured in oozie, and some jobs are signal free and some are dependent on signals. We have given signal free jobs start time 2AM and the jobs were starting firing during that time. From last one month we have noticed those signal free jobs started delaying by 1 hour. We are not sure why is it happening.
does any one have idea on this, why oozie jobs started executing with delay.
if the system time changes for day light saving time, does the output of gettimeofday() is changed?
Note that Unix systems do not "change the system time" with daylight saving time -- it is all handled when programs want to print the current time, typically using localtime(3).
The seconds since the epoch keep counting monotonically even when local governments decide to change the clocks on our walls.
UNIX/linux computers use UNIX time which is more or less UTC i.e. Greenwich London. Only when you print or display the time does it get converted into localtime. This is done by consulting the TIMEZONE setting.
On linux the timezone database is in /usr/share/zoneinfo - the current timezone is defined in /etc/localtime. This file handles daylight savings time so the conversion when you print is handled cleanly. Note that the timezone is usually defined for geographical and political reasons.
Your timezone database will be periodically updated as various governments and/or local councils decide to change their timezone or daylight savings time settings. Recently Samoa skipped a complete day at the end of 2011. Provided the TIMEZONE files on any local Samoan UNIX/Linux computers had been updated beforehand then no problems would have been encountered.
Other considerations are NTP time synchronisation and leap seconds. If you run an NTP client to keep your computer in sync then this 'slews' the internal computer clock by either speeding it up or slowing it down - note that it is usually a bad idea to set the time manually using the 'date' or 'rdate' commands as this can cause a jump in time and may affect software that uses timeouts of some kind. (A recent example was an Asterisk PABX that I had to maintain).
If you are going to change the time manually use the ntpdate command and define an upstream NTP server.