I've been running airflow 1.9.0 and using dynamically generated schedule intervals.
Simply put, I take a US/Eastern timestamp from some config file, get the current system timezone (can be either EDT or EST), and convert it to UTC time then to a cron expression.
For example, if I launch the dag today (2018-07-23, EDT) and my input is 6AM US/Eastern, it will result in a dag whose schedule interval is 10AM UTC or 0 10 * * 1-5.
My question is:
if I leave the dag running on a daily basis, will its schedule automatically update to 0 11 * * 1-5 in November, when daylight savings ends?
I specifically want to avoid using tz-aware datetimes in scheduling these dags, that's why I came up with this hacky way of timestamp conversion.
What library or code are you using to do the conversion between your Eastern timestamp and generate the cron expression? I think answering this part of your question is dependent on that information.
Anyway, this idea kind of sounds like a code smell to me. While it would technically work, assuming you library supports that use case correctly and that the timezone library is kept up-to-date, I believe you're better off going with the standard route of determining the crontab schedule you want upfront and using that consistently.
It's also a best practice to not use local time zone, for example, in the case where you move your server from Eastern to Pacific or operate multiple servers in different timezones — using UTC everywhere keeps it simple as you scale up.
Since UTC does not have daylight saving time, this will help you avoid things like DST bugs that you'd otherwise have to address if not using UTC.
Additionally, the official Airflow docs recommend against using naive datetimes:
Because Airflow uses time-zone-aware datetime objects. If your code creates datetime objects they need to be aware too.
...
Although Airflow operates fully time zone aware, it still accepts naive date time objects for start_dates and end_dates in your DAG definitions. This is mostly in order to preserve backwards compatibility.
...
Unfortunately, during DST transitions, some datetimes don’t exist or are ambiguous. In such situations, pendulum raises an exception. That’s why you should always create aware datetime objects when time zone support is enabled.
https://github.com/apache/incubator-airflow/blob/master/docs/timezone.rst
Can you elaborate on your use case for using naive datetimes vs timezone aware datetimes? I'd be happy to add more specific advice about that.
Related
I have a request to alter current columns which are type of 'time' and instead of capturing just time I need to capture so called "utc time".
My idea is to create a fixed codelist with all timezones, and then to reference it to a appropriate table as FK.
My questions are:
Can column of a type 'time' hold also an information regarding time zones (utc, for example 15:00:00 +2 (gmt + 2)) and if not, could you suggest me another type for that column?
Should I maybe need to separate it into two columns? For example: [15:00:00] - StartTime, [+2:00] - UtcOffset
EF Insert: When I do inserting to the db, for that particular column, should I convert my DateTime object to for example DateTimeOffset?
Thanks in advance.
From the comments in your question, it sounds like you are building an appointment scheduling system. I'll base my answer on that, because your specific questions aren't quite aligned to the scenario you described.
First, it's important to understand that the relationship between a time zone and an offset is a one-to-many relationship. One time zone can have multiple offsets. In other words, a time zone is not an offset, but rather a time zone has an offset.
A time zone represents a geographic region where the local time is the same throughout. It is identified by a string ID, such as "America/Los_Angeles" (an IANA time zone ID) or "Pacific Standard Time" (a Windows time zone ID). In .NET, you will use them on the TimeZoneInfo object with the Id instance property or methods like FindSystemTimeZonesById.
An offset is like -07:00 or +05:30 or even +13:45. Any given offset applies only at a particular date and time. For example, in America/Los_Angeles, either -08:00 or -07:00 apply depending on whether daylight saving time is in effect at a given point in time. Keep in mind that DST is not the only reason for offsets to be different - many time zones have changed their standard time at some point in their history.
Also, it's called an offset because it is deviated from UTC by a certain amount. UTC itself always has a zero offset, delineated either by +00:00 or sometimes by Z. It's similar to GMT, except in how it is defined. UTC applies universally, everywhere. GMT technically applies only on the prime meridian. They both refer to the zero offset. You should prefer to say "UTC" in most cases.
Next, you should separate your application logic between future scheduling and present/past record keeping.
Present/past is the easier of the two. Since the moment in time has actually occurred, the local time and its offset from UTC is fixed forever. You can either store the local time and its offset in a single .NET DateTimeOffset structure (mapped to a datetimeoffset field in SQL Server). In other words, you can simply store 2021-07-27T12:00:00-08:00.
Note that you could instead store the equivalent UTC date and time, which would be 2021-07-27T20:00:00+00:00. However you've then lost the local time, and thus would need to convert back using the original time zone if you wanted to see that time. Some people prefer that, but I think it's more useful to store the original value.
For future scheduling, the situation is a bit different. Consider that the offset might not be the same for one appointment as it will be for the next appointment in the same time zone. Also consider that the definitions for which offset apply might change in between the time you schedule the appointment and when it comes around. (The likelihood of that increases the further out you schedule.)
Thus, for each location you should not store an offset, but rather a time zone identifier. Add a TimeZoneId to your object that stores each location (or each appointment depending on your model schema). Use TimeZoneInfo.GetSystemTimeZones to list the available time zones. The DisplayName property can be shown to your user, and the selected Id property gets assigned to the TimeZoneId.
Next you have to consider if you are scheduling a single appointment or creating a recurring appointment pattern.
For a single appointment you simply need the local date and time for that appointment. You can use a .NET DateTime struct (use datetime2 in SQL). Don't apply an offset, and don't convert to UTC. Just store the information provided.
For recurring appointments, you need to think through the information provided and store exactly what is given. For example, if the appointment is at 10:00 every other Tuesday, you'll need to store "10:00", "Tuesdays" and "2 week intervals". The data types for each will vary depending on how you choose to store and apply them. For example, you might use a time type for 10:00, but you could store the other values using integers. Appointments of different patterns could get stored in different ways.
Alternatively, some like to store patterns using a string containing a CRON expression. You can google that for more details.
Now you have everything you need to both schedule an appointment and record that appointment after it happens. But there's one part missing - you'll likely want some table of upcoming appointments that are easily queryable. For that, you've got a few options:
You can create a separate table via a background job of some kind. Periodically it would query all the appointments, use their information to compute the next upcoming appointment time, and insert it. You can store that in a DateTimeOffset, either as local time or as UTC. (SQL Server will always compute indexes on the equivalent UTC time either way.)
You could just add another field to your appointments that shows the next actual appointment time. You can then compute the next upcoming appointment whenever the appointment is created or updated, or when that appointment occurs, and update the table accordingly.
With either approach keep in mind that you will want to periodically check for time zone data updates (either via Windows Update or keeping the tzdata package current on Linux). You will also want to periodically re-compute future appointment times, in case that time zone data has been modified in a way that affects the appointments.
If all of this sounds super complicated - sorry, but it is. Doing scheduling worldwide across time zones right is challenging. If you want it simpler, you might want to look into a pre-made solution such as Quartz.NET, which you can integrate into your application.
I want to fetch date and time, in different timezone using SQLITE. Say, I get the value as, 20160118T010856 in input, but I require it to change to Australian, England and other timezone.
What I am currently able to do is :
select datetime('2016-01-18T04:13:39','localtime');
Which only provides me for my current local time. Kindly help.
As far as I know this is not possible in sqlite. Your best bet is to store the datetime as UTC and do the conversion outside the database.
From the documentation you could apply an offset(modifier) to a date and time, but it doesn't take Daylight Savings Time into account.
We have an ASP.NET website and an SQL database hosted in US. Whenever I use the function Now() in VB.NET and getdate() in SQL, I get the US' current time. The problem is, the client is in the Philippines which is on GMT+8 Time Zone. My question is, is there any way I can set the Time Zone of a specific database and website so that when I use the functions, I'll get the Philippine's current time? How do you deal with this? As much as possible, we don't want to do subtraction or addition to the result date of the functions since in the future, clients will be from other country. It will give us headache updating the codes if we do that.
Thank you in advance!
Given that your clients may be in different time zones, you should store a timezone for the clients, that they (or you) can set as a preference for their account. Store all dates+times as UTC, and then convert to their timezone when displaying results in your interface.
This question has already been addressed to a great extent in the following question:
How to work with time zones in ASP.NET?
Follow-up:
Unfortunately, the SQL server date is a system-level setting, so it's not really something that can be manipulated on a per-session basis. It sounds like you will need to make some code changes, but you can isolate them.
Do you have a session-level variable which contains the client time zone offset? If not, create one.
Create a small date/time utility class.
In the utility class, provide 3 methods to:
(1) get the current date/time (offset to the client's time zone)
(2) pass in a database date/time to return the time offset for the client's TZ.
(3) pass in a time from the client to subtract out the client's TZ difference.
You will have to make code changes, but you can probably use those utility functions to wrap inputs and outputs everywhere, centralizing the logic. Microsoft has a page about mis-steps to avoid when using the DateTime class and manipulating time zones:
http://msdn.microsoft.com/en-us/library/ms973825.aspx#datetime_topic1a
I'm looking for an easy way to get UTC times and date/time information in vbscript, even while specifying a time zone. I need to be able to get the current time in a time zone of my choice.
I have seen postings on the web for functions that determine DST, but I'd rather not use something I'd have to update if DST or time zones switched--so using a standard library would be ideal. I don't have the option of using a web service with this application. I'm open to other ideas, though.
If you have access to a database you could use a database query to determine time zones.
For MSSQL, see this way of doing it
For TSQL, see this stack overflow question
I am working on cleaning up a bug in a large code base where no one was paying attention to local time vs. UTC time.
What we want is a way of globally ignoring time zone information on DateTime objects sent to and from our ASP.NET web services. I've got a solution for retrieve operations. Data is only returned in datasets, and I can look for DateTime columns and set the DateTimeMode to Unspecified. That solves my problem for all data passed back and forth inside a data set.
However DateTime objects are also often passed directly as parameters to the web methods. I'd like to strip off any incoming time zone information. Rather than searching through our client code and using DateTime.SpecifyKind(..) to set all DateTime vars to Undefined, I'd like to do some sort of global ASP.NET override to monitor incoming parameters and strip out the time zone information.
Is such a thing possible? Or is there another easier way to do what I want to do?
Just to reiterate -- I don't care about time zones, everyone is in the same time zone. But a couple of users have machines badly configured, wrong time zones, etc. So when they send in July 1, 2008, I'm getting June 30, 2008 22:00:00 on the server side where it's automatically converting it from their local time to the server's local time.
Update: One other possibility would be if it were possible to make a change on the client side .NET code to alter the way DateTime objects with Kind 'Undefined' are serialized.
I have dealt with this often in many applications, services, and on different platforms (.NET, Java, etc.). Please believe me that you do NOT want the long term consequences of pretending that you don't care about the time zone. After chasing lots of errors that are enormously difficult and expensive to fix, you will wish you had cared.
So, rather than stripping the time zone, you should either capture the correct time zone or force a specific time zone. If you reasonably can, get the various data sources fixed to provide a correct time zone. If they are out of your control, then force them either to the server's local time zone or to UTC.
The general industry convention is to force everything to UTC, and to set all production hardware clocks to UTC (that means servers, network devices like routers, etc.). Then you should translate to/from the user's local time zone in the UI.
If you fix it correctly now, it can be easy and cheap. If you intentionally break it further because you think that will be cheaper, then you will have no excuses later when you have to untangle the awful mess.
Note that this is similar to the common issue with Strings: there is not such thing as plain text (a String devoid of a character encoding) and there is no such thing as a plain (no time zone) time/date. Pretending otherwise is the source of much pain and heartache, and embarrassing errors.
OK, I do have a workaround for this, which depends on the fact that I only actually need the Date portion of the DateTime. I attach this property to every Date or DateTime parameter in the system
<XmlElement(DataType:="date")>
This changes the generated wsdl to have the type s:date instead of s:dateTime. (Note that simply having the type of the .NET method parameter be a Date rather than a DateTime did NOT accomplish this). So the client now only sends the date portion of the DateTime, no time info, no time zone info.
If I ever need to send a Date and Time value to the server, I'll have to use some other workaround, like making it a string parameter.
I've had issues with the time zone information as well. The problem is I'm already providing the datetime fields in UTC. Then the serialization occurs and the local offset becomes part of the date/time. The dates/times for our vendor in a different timezone were pretty messed up. I got around this problem by using the tsql convert function on the datetime fields in my select statement I used to populate my datasets. This converted the fields to a string variable, which translates nicely to a datetime value automatically on the client side. If you just want to pass the date, you can use the 101 code to provide just the date. I used 126 to provide the date and time exactly how it appears in my database columns, with the timezone information stripped out.