I have a Redshift data table where all time values are stored in CST and I convert the time values to the respective timezone based on the zip code (location).
While I do that, I understand that all time values are in Standard time and hence my function usage is
CASE WHEN **** convert_timezone('CST', 'EST', time_column)
WHEN **** convert_timezone('CST', 'MST', time_column)
....
END
This may not be applicable once we enter into Daylight Savings time. How can I take care of this such that I do not modify the SQL query again in 2018 March and in future?
Don't use time zone abbreviations. The are somewhat ambiguous, and can only refer to one aspect of the time zone. Instead, use a full IANA time zone identifier, such as America/Chicago for US Central time.
This is explained well in the Redshift docs:
Using a Time Zone Name
If you specify a time zone using a time zone name, CONVERT_TIMEZONE automatically adjusts for Daylight Saving Time (DST), or any other local seasonal protocol, such as Summer Time, Standard Time, or Winter Time, that is in force for that time zone during the date and time specified by 'timestamp'. For example, 'Europe/London' represents UTC in the winter and UTC+1 in the summer.
As far as the "...based on the zip code" part of your question, understand that not every ZIP code is locality-based. There are also technical assignments, overseas APO/FPO addresses, US territories, and other edge cases. Additionally, some zip codes may straddle more than one time zone.
A better approach, when possible, is to:
Get an approximation of latitude/longitude coordinates - using a variety of techniques depending on your source data. For example, geocoding APIs can take a street address and give a lat/lon.
Then determine the time zone identifier for that location, using one of the techniques listed here.
Related
I always had conceptual troubles with date values being represented as objects without time (SQL for instance has the type DATE beside DATETIME). The same thinking often creates issues in languages that always store dates along with a time (e.g. C#'s DateTime). Suddenly timezone bugs pop up and the developers insist that all that timezone hullabaloo can be ignored, because their DateTime is actually "just a date". But what does that even mean?
The way I see it, such a "plain date" can be interpreted in only two ways:
From the perspective of points in time (i.e. "instants") one might infer that "2019-09-17" means the start (end) of the day, which is "2019-09-17 00:00:00" ("2019-09-17 23:59:59"). Hence timezone information is relevant and we are not talking about "just a date".
The other option is that the date is supposed to mean the whole day, so we are actually talking about a timespan from "2019-09-17 00:00:00" to "2019-09-17 23:59:59" and timezone information cannot be ignored either.
I argue that all use cases for "plain dates" fall in either of the two categories and that time/timezone can only be ignored if the software will only run on and communicate with systems configured with the exact same timezone.
Can someone provide a counter example or a third interpretation?
The "holiday" is an example of a date without a time, and even without a time zone.
Most holidays are religious in nature, and I don't want to start a religious war, or get sidetracked by minutiae. So as an example holiday, I'm going to pick Earth Day: April 22 of each year in the civil calendar.
If you write software that deals with holidays, said software isn't necessarily dealing with an instant in time across the entire planet (though it could do that in addition to tracking holidays). Instead it carries the concept that maps April 22 of each year to Earth Day. And it is irrelevant that some people will celebrate Earth Day hours before or after other people.
If a client wants to know exactly when Earth Day starts/ends for him, then he can translate "the date" into specific instants of time using his particular time zone. The need for that translation does not diminish the value of the date (say 2020-04-22).
C++20 will have several types to represent a "timezone-less date" such as 2020-04-22 using different data structures:
local_days // day 18374 == April/22/2020
year_month_day // year 2020, month 4, day 22 == April/22/2020
year_month_weekday // year 2020, month 4, 4th Wed == April/22/2020
We have a database of cities with its geo coordinates. Once we filled it with corresponding time zones using tzworld. User sets location including city, city has time zone - here how we know user's timezone (we need to render date and time on server). But time zones are being changed: some new are appearing, some old are being removed.
Is there any best practices or tools to handle that kind of changes?
I.e. there is a city Foo with time zone Foo/Bar. One day tzdata was changed, and Foo/Bar was split into Foo/Old_Bar and Foo/New_Bar time zones with the same UTC offsets. We still have Foo/Bar in our db. Actually, it's a BC break, but it's ok since, say, we can handle those BC breaks. But then tzdata was changed again, and now Foo/New_Bar has different offset. And here comes troubles. Some users from Foo city see wrong local time since that moment.
Just to be sure you understand me right: it's not about DST, it's about the fact that time zones (their names) are being changed.
As far as I can see, wee need a kind of machine-readable tzdata diff. Like
split: Foo/Bar Foo/Old_Bar,Foo/New_Bar
move: Foo/New_Bar -05:00
This issue makes me feel that storing time zones is a bad idea. Is there a better one?
With specific regard to the IANA/Olson TZ database, the location identifiers do not change once established. The history of each identifier is always consistent for that location.
However, if you are using tz_world or some other map source to determine the time zone for some other location - one that doesn't necessarily have it's own identifier, then yes - it's possible that a zone split will cause the zone to change. Though, when it does, the new zone should be consistent with the old zone, up to the point of the change.
As a real world example, consider America/Fort_Nelson, which was added in tzdb 2015g for Fort Nelson, British Columbia, Canada, and the surrounding region of the Northern Rockies Regional Municipality. Previously, this area would have been resolved to America/Vancouver, but the zone was split due to their March 2015 time change. The tz_world maps were updated on November 7, 2015 to account for this change.
If you had previously resolved a user in Fort Nelson to America/Vancouver, then they will have incorrect times from November 1st, 2015 forward, as that's when Vancouver switched back to UTC-8, while Fort Nelson remained at UTC-7.
If you update to the latest tzdb and tz_world, you can use the original information to re-determine the time zone - which would now be America/Fort_Nelson.
The new time zone will accurately reflect all of the same information as Vancouver before the split, and the correct information for Fort Nelson after the split.
All of this should just work, assuming you update time zones after each update of tz_world, and recalculate future events after updating the tzdb.
The question remains, how do you know which zones have split and changed so you don't have to recalculate everything? For a small amount of data, you might as well recalculate everything. But for larger datasets, this might be impractical. Unfortunately, there's no machine-readable standardized format for the differences. I believe this has been talked about before in the tz discussion list, but I can't find it at the moment. You can ask there if you like.
Currently the only way is to manually read the release notes of each update. You can find them in the tz-announce list archives (or subscribe to the list for future updates). You can also find them in the NEWS file of any given release. You'll also want to review the history of the tz_world shapefile, which is on that web site.
Also, recognize that time zone IDs will never be removed from the tzdb. A split may create a new zone (Foo/New_Bar), but the original zone will remain (Foo/Bar, not Foo/Old_Bar). If a zone is determined unnecessary, its Zone entry might be replaced with a Link entry, but it will never be removed entirely.
I imagined there would be more literature on this, but I'm having trouble finding any. I have a lot of non-algebraically-aggregatable time series data (that is to say, points for which no function exists that I could use to aggregate them to a higher granularity-- stuff like unique active users, unique contributors, etc... where knowing the amount I had every minute of some hour does not tell me how many I had total during the hour). Currently, I'm just storing and presenting all of this data in UTC. The problem is that many of my clients find this confusing-- understandably so. Because the data is non-algebraically-aggregatable, there's no way to get from UTC data for 1 day midnight- midnight to, say, PST data from midnight to midnight. Recalculation would need to be done from raw data.
So:
Recalculation from raw data is prohibitively expensive for some complicated analytics graphs
We could store all data for all time zones, but this would increase the amount of data we store x24.
All of that said, how do other people deal with this issue? Here's how Google Analytics does it, but this seems insufficient for my use case because I know if I open the multiple timezone can of worms, clients will ask for more than one. This will also take a lot of work that doesn't seem worth the effort as just adding timezone support won't be extremely noticeable or a huge win. What I'm really hoping for is some clever design solution that just presents the UTC data in some intuitive enough way that it's no longer confusing for people in other timezones. Has anyone dealt with similar problems and come upon a solution I'm missing?
First of all, you should recognize that there a lot more than 24 time zones. In order to accurately take into account how people actually use time worldwide, you should be using IANA time zones, of which there are over 500. See also Wikipedia and the timezone tag wiki.
If you are dealing with individual points (discreet timestamps), then you can certainly convert from UTC to any time zone you wish, on the fly as you render your graph. You just need to also keep in mind that the range of data you query will also need to be translated to that time zone.
But if you are talking about aggregating data by the "day" of a specific time zone, then there is no magic bullet. You will need to decide ahead of time which time zones you want to support and calculate each one separately. When you do this, recognize that it's not just the view that's changing. Since the day boundaries are different for each time zone, then the data for each time zone could potentially have very different daily totals.
You should also be aware that not every day has 24 hours. If the day happens to be the date of a daylight saving time transition, it could have 23, 23.5, 24.5, or 25 hours. This could potentially affect how you draw your graph.
One approach you might consider is to be time zone ignorant in your aggregations, rather than using UTC or any specific time zone. Of course this depends heavily on the context of your data, but it is appropriate in certain circumstances. For example, on an invoice, you might care less about the specific timestamps, and more about which calendar date the invoice was assigned to. In that case, once a date is assigned, you would just aggregate on that date. Even if the company operates over multiple time zones, you wouldn't care about that in aggregate.
As far as some clever design that abstracts this from the user, I'm afraid I haven't seen much. The only two choices you really have are timezone-adjusted aggregations (UTC or otherwise), and time zone ignorant aggregations for calendar-date contexts.
We had similar issues to roll up the data for Generation in renewable. We went with three options User / Farm / UTC.
If user selects USER then all the data would be based on his browser Time zone. And Yesterday meant 24 hours till last mid night in user local time.
Similarly if it was Farm, then we take the Farm local and derive the same.
UTC is standard similar to what you have implemented.
I'm writing a geotagging app and running into headaches with timezones. Basically, my app has the following data:
Images with local timestamps (i.e. relative to a timezone)
GPS track files consisting of entries using UTC timestamps
My problem: I need a way to get all data that belongs to a give day, based on the timezone where the data was acquired. For the images, that is easy (I ask the timezone from the user upon import and save it in the EXIF data), but I'm not sure how to do it for the GPS tracks (there usually are multiple tracks per day, and assigning them timezones is not easy for the user when importing data that spans sever days and timezones). I can think of two possible solutions:
Use a heuristic based on the fact that the tracks are recorded at the same time and place as the images - but there can be tracks before a day's first image or after its last one that still need to be included - I'm not sure how to realiably handle such edge cases
Determine the timezone from the GPS coordinates - this would be an ideal solution, but is there an open source library that does this (ideally one that works offline)?
The heuristic method I don't think will work well.
Firstly always store times as UTC and timezone of origin, otherwise time is less meaningful.
After some thought I think that it would be sufficient to resolve down to the country code and from that lookup the timezone.
Depending on how much detail you want I think GeoTog may help you to locate a city and therefore a country from a lat/long (although it will need changing to work the other way).
If not that then Gisgraphy will work with the larger GeoNames database. You could use the web service or extract the data.
If none of these are good enough then I think you'll need to get a some GIS data, possibly boundaries from VMAP0 and process it into polygons or something searchable.
Option two: you could start by checking this site: http://www.twinsun.com/tz/tz-link.htm
Option one (less complicated, but I am not sure I accurately understand your need...)
So you have as input:
A target day defined in a known timezone TZ, starting at t0 and ending at t1 (excluded)
Images with timestamps ti in the same timezone TZ (is this hypothesis true?)
GPS tracks with UTC timestamps tg which can span over several time zones
We also know that there is at least one GPS track for each image.
Here's something that should work:
Convert your target day into UTC. You get the values t0/UTC and t1/UTC
Convert images' timestamps into UTC (you get ti/UTC from known ti/TZ)
process image if (t0/UTC <= ti/UTC < t1/UTC) i.e. it was taken during your target day.
find a GPS track including ti/UTC (no problem since tracks are timestamped in UTC), and then the closest timestamp within the list of points in this track. This point is the most likely position of your image.
I want to know if there can be 2 or more GMT timezones for one city or state. I know there can be more then one GMT timezone for a country, but not sure if it's for state and city too. Share your knowledge please.
Interpreting the question to mean 'are there any cities which are in more than one time zone', then the answer is 'yes'. And there are American states with multiple time zones (Indiana and Arizona being two of them).
There has been recent discussion on the TZ mailing list about the area of China known as Xinjiang, which has a mixed population of Han Chinese and of Uyghurs. It seems that the Han use the standard Chinese time zone (Asia/Beijing), but the Uyghurs often use a local time zone. This is now encapsulated in the Olson database, with the name Asia/Urumqi for the Uyghur time zone.
So, for example, the zone.tab file in tzdata2010b.tar.gz, available from ftp://elsie.nci.nih.gov/pub/tzdata2010b.tar.gz (the code is ftp://elsie.nci.nih.gov/pub/tzcode2009t.tar.gz). There is an extensive description of how and why the change was made in the asia file.
Note that the Olson (Time Zone) database is now (2016-09-19) available from IANA at https://www.iana.org/time-zones rather than from NIH. You can get the current release easily enough; getting historical releases may be harder.
Yes, time zones really do change 20 times a year around the world, and sometimes at essentially no notice (that is, the government legislates the changes only a day or two before the change).
#basit asks:
Wow about the 20 times a year around the world. I'm trying to log the timezone for latitude and longitude, so now my question would be, how long should I log the data for? 6 months? 1 month? 2.. 3..?
And also, how long does it take for daylight savings to change in a year, because I need to log timezone with daylight saving and refresh the data after certain given period.
What I mean is that during the course of 2009, there were 20 issues of the time zone database, because of changes in rules in at least that many places. However, any given country usually only changes their rules once - though with Argentina, different states were changing their rules at different times and compounding the problems.
I'm not clear that we have enough information to tell you how long to log the data for. I'd be inclined to say at least 12 months, but it depends what you are going to do with it. At one level, all you need to do is keep up with the Olson database - that will tell you the time zone rules for essentially everywhere in the world. If you are interested in tracking the time zones of your visitors, then you can keep the data for as long as you like. Since not everyone uses the canonical Continent/City notation for their time zone (I tend to use the older US/Pacific notation, for instance - which is still supported, but is equivalent to America/Los_Angeles). The classical notations such as TZ=EST5EDT are ambiguous; both the USA and Australia have timezones that use EST as an abbreviation, and the dates when the switch between standard and daylight saving time occurs varies (witness the mass of data in the Olson database).
You also ask 'how long does it take for a time zone to change'. I'm not sure what you mean. In terms of 'when the clocks change (between standard and daylight saving time)', it is 'instantaneous'; one second it is one time zone offset; the next second it is the other. If you mean 'how long does it take for governments to change their mind', it varies radically. For example, both Europe and the USA have relatively fixed rules that change every few years; the rule in the USA had been stable for about 20 years, then they changed the rules about 3 years ago. Europe is similar. On the other hand, some countries change their rules yearly. My impression is that some of the Islamic countries adjust when they switch between standard and daylight saving time (or vice versa) depending in part on when Ramadan falls - if the change would occur during Ramadan, then they bring it forward, or delay it, so that the rule does not change during Ramadan. Other countries have different reasons for the brinksmanship that goes on - maybe it is the political equivalent of a release deadline. So it may take quite a while for people to decide what the 'final' (meaning 'next edition') of the rules will be for a given year.
The web site http://worldtimezone.com/ does a pretty good job of keeping track of most of these idiosyncracies.
I think you mean "Can one city or state span two time zones?". Yes. Mexico Beach, FL sits on the border between CST and EST with parts of the town in both time zones.
As for how you could tell a computer that, no idea.
There is only one gmt for the whole world. As for timezones, see here, showing variation of observance e.g. within Kansas.
Any arbitrary jurisdiction may have multiple timezones, though the majority do not.
Have a look at http://www.worldtimezone.com/faq.html