I have a TIMESTAMP column in a teradata table. I want to convert the timestamp to epoch value. Can someone shed some light on how to do this.
This is a SQL UDF I wrote a few years ago.
If you don't have access rights to create a function, you mighty ask you dab or simply cut & paste the calculation.
/**********
Converting a Timestamp to Unix/POSIX/epoch time
Unix time: Number of seconds since 1970-01-01 00:00:00 UTC not counting leap seconds (currently 34 in 2010)
The maximum range of Timestamps is based on the range of INTEGERs:
1901-12-13 20:45:52 (-2147483648) to 2038-01-19 03:14:07 (2147483647)
Simply change to BIGINT to cover the full Teradata date range
20101211 initial version - Dieter Noeth
**********/
REPLACE FUNCTION TimeStamp_to_UnixTime (ts TIMESTAMP(6))
RETURNS INT
LANGUAGE SQL
CONTAINS SQL
DETERMINISTIC
SQL SECURITY DEFINER
COLLATION INVOKER
INLINE TYPE 1
RETURN
(CAST(ts AS DATE) - DATE '1970-01-01') * 86400
+ (EXTRACT(HOUR FROM ts) * 3600)
+ (EXTRACT(MINUTE FROM ts) * 60)
+ (EXTRACT(SECOND FROM ts))
;
Reversing the calculation:
/**********
Converting Unix/POSIX/epoch time to a Timestamp
Unix time: Number of seconds since 1970-01-01 00:00:00 UTC not counting leap seconds (currently 34 in 2010)
Also working for negative numbers.
The maximum range of Timestamps is based on the range of INTEGERs:
1901-12-13 20:45:52 (-2147483648) to 2038-01-19 03:14:07 (2147483647)
Simply change to BIGINT to cover the full Teradata date range
20101211 initial version - Dieter Noeth
**********/
REPLACE FUNCTION UnixTime_to_TimeStamp (UnixTime INT)
RETURNS TimeStamp(0)
LANGUAGE SQL
CONTAINS SQL
DETERMINISTIC
SQL SECURITY DEFINER
COLLATION INVOKER
INLINE TYPE 1
RETURN
CAST(DATE '1970-01-01' + (UnixTime / 86400) AS TIMESTAMP(0))
+ ((UnixTime MOD 86400) * INTERVAL '00:00:01' HOUR TO SECOND)
;
It's is easier in TD14 using TO_TIMESTAMP(UnixTime), but this is restricted to the INTEGER range.
Related
I want to calculate the difference between two columns containing datetime stamps in db browser SQLite, I want the answers in minutes, and it keeps returning "Null". Please what could be the reason and how can I solve it?
I tried using this;
SELECT
started_at,
ended_at,
(strftime('%M','ended_at') - strftime('%M','started_at'))as duration
FROM citi1;
You have 'started_at' and 'ended_at' which are string literals and not identifiers and SQLite returns null when you use them in strftime().
But, even if you remove the single quotes you will not get the timestamp difference, because subtracting only the minutes parts of 2 timestamps does not return their difference.
For example, the difference that you would get for started_at = '2022-03-31 13:15:00' and ended_at = '2022-03-31 14:00:00' would be -15 (= 0 - 15).
Use strftime('%s', some_date) which returns the number of seconds since 1970-01-01 00:00:00 for both timestamps, subtract and divide by 60 to get the correct difference in minutes:
SELECT started_at, ended_at,
(strftime('%s', ended_at) - strftime('%s', started_at)) / 60 AS duration
FROM citi1;
See the demo.
I am working locally with an sqllite DB. I have imported some records from teradata where there was a date field in the format of 'YYYY-MM-DD'. When i imported the records the date switched from a date to a number. I know this is a feature of sqllite and that one can access it via date(sqllite_date) when selecting it in a where clause.
My problem is that the dates now appear to be a bit odd. For example the year appears to be negative.
Is there anyway to recover this to the correct format?
Below is an example of converting a number in the database into a date
SELECT date(18386)
# -4662-03-28
SELECT datetime('now')
# 2021-02-11 10:41:52
SELECT date(sqllite_date) FROM mydb
# Returns -4662-03-28
# Should return 2020-05-04
I am very new to this area so apologies if this is a basic question. Thank you very much for your time
In SQLite you can store dates as TEXT, REAL or INTEGER.
It seems that you stored the dates in a column with INTEGER or REAL affinity.
In this case, if you use the function date(), it considers a value like 18386 as a Julian day, meaning the number of days since noon in Greenwich on November 24, 4714 B.C.
This is why date(18386) returns 4662-03-28B.C.
But I suspect that the date values that you have are the number of days since '1970-01-01'.
In this case, 18386 days after '1970-01-01' is '2020-05-04'.
So you can get the dates in the format YYYY-MM-DD if you add the value of your column as days to '1970-01-01':
SELECT date('1970-01-01', datecolumn || ' day') FROM tablename
Or by transforming your date values to seconds and treat them as UNIX time (the number of seconds since '1970-01-01 00:00:00 UTC'):
SELECT date(datecolumn * 24 * 3600, 'unixepoch') FROM tablename
Replace datecolumn with the name of your column.
I have data in Google BigQuery that looks like this:
sample_date_time_UTC time_zone milliseconds_between_samples
-------- --------- ----------------------------
2019-03-31 01:06:03 UTC Europe/Paris 60000
2019-03-31 01:16:03 UTC Europe/Paris 60000
...
Data samples are expected at regular intervals, indicated by the value of the milliseconds_between_samples field:
The time_zone is a string that represents a Google Cloud Supported Timezone Value
I'm then checking the ratio of the actual number of samples compared to the expected number over any particular day, for any single day range (expressed as a local date, for the given time_zone):
with data as
(
select
-- convert sample_date_time_UTC to equivalent local datetime for the timezone
DATETIME(sample_date_time_UTC,time_zone) as localised_sample_date_time,
milliseconds_between_samples
from `mytable`
where sample_date_time between '2019-03-31 00:00:00.000000+01:00' and '2019-04-01 00:00:00.000000+02:00'
)
select date(localised_sample_date_time) as localised_date, count(*)/(86400000/avg(milliseconds_between_samples)) as ratio_of_daily_sample_count_to_expected
from data
group by localised_date
order by localised_date
The problem is that this has a bug, as I've hardcoded the expected number of milliseconds in a day to 86400000. This is incorrect, as when daylight saving begins in the specified time_zone (Europe/Paris), a day is 1hr shorter. When daylight saving ends, the day is 1hr longer.
So, the query above is incorrect. It queries data for 31st March of this year in the Europe/Paris timezone (which is when daylight saving started in that timezone). The milliseconds in that day should be 82800000.
Within the query, how can I get the correct number of milliseconds for the specified localised_date?
Update:
I tried doing this to see what it returns:
select DATETIME_DIFF(DATETIME('2019-04-01 00:00:00.000000+02:00', 'Europe/Paris'), DATETIME('2019-03-31 00:00:00.000000+01:00', 'Europe/Paris'), MILLISECOND)
That didn't work - I get 86400000
You can get the difference in milliseconds for the two timestamps by removing the +01:00 and +02:00. Note that this gives the difference between the timestamps in UTC: 90000000, which is not the same as the actual milliseconds that passed.
You can do something like this to get the milliseconds for one day:
select 86400000 + (86400000 - DATETIME_DIFF(DATETIME('2019-04-01 00:00:00.000000', 'Europe/Paris'), DATETIME('2019-03-31 00:00:00.000000', 'Europe/Paris'), MILLISECOND))
Thanks #Juta, for the hint on using UTC times for the calculation. As I'm grouping my data for each day by a localised date, I figured out that I can work out milliseconds for each day by getting the beginning and end datetime (in UTC), for my 'localised' date, using the following logic:
-- get UTC start datetime for localised date
-- get UTC end datetime for localised date
-- this then gives the milliseconds for that localised date:
datetime_diff(utc_end_datetime, utc_start_datetime, MILLISECOND);
So, my full query becomes:
with daily_sample_count as (
with data as
(
select
-- get the date in the local timezone, for sample_date_time_UTC
DATE(sample_date_time_UTC,time_zone) as localised_date,
milliseconds_between_samples
from `mytable`
where sample_date_time between '2019-03-31 00:00:00.000000+01:00' and '2019-04-01 00:00:00.000000+02:00'
)
select
localised_date,
count(*) as daily_record_count,
avg(milliseconds_between_samples) as daily_avg_millis_between_samples,
datetime(timestamp(localised_date, time_zone)) as utc_start_datetime,
datetime(timestamp(date_add(localised_date, interval 1 day), time_zone)) as utc_end_datetime
from data
)
select
localised_date,
-- apply calculation for ratio_of_daily_sample_count_to_expected
-- based on the actual vs expected number of samples for the day
-- no. of milliseconds in the day changes, when transitioning in/out of daylight saving - so we calculate milliseconds in the day
daily_record_count/(datetime_diff(utc_end_datetime, utc_start_datetime, MILLISECOND)/daily_avg_millis_between_samples) as ratio_of_daily_sample_count_to_expected
from
daily_sample_count
Here are my table's columns :
Time | Close | High | Low | Open | pairVolume | Trades | Volume
I would love to have my data group by range of time.
Now the tricky part is that this range is custom (it's a user input which could very well be grouping by 10 minutes, 2 hours, or even 5 days)
My time field is stored in millisecond since epoch.
Solution I found for now which I'm uncertain about :
SELECT time + (21600000 - (time%21600000)) as gap, count(time)
FROM price_chart
WHERE time >= 1517418000000 and time <= 1518195600000
GROUP BY gap
21600000 is 6 hours in milliseconds
time is time since epoch
Yes, it works.
Putting some numbers into excel with your formula below, it works for me. Your gap value will be returned as the top end of each time range grouping.
SELECT time + (21600000 - (time%21600000)) as gap ...
Using the below:
SELECT time - (time%21600000) as gap_bottom ...
Would return you the bottom end of each time range grouping. You could add this as an additional calculated column and have both returned.
EDIT / PS:
You can also use the SQLite date formatting functions after dividing 1,000 milliseconds out of your epoch time and converting it to the SQLite unixepoch:
strftime('%Y-%m-%d %H:%M:%S', datetime(1517418000000 / 1000, 'unixepoch') )
... for ...
SELECT strftime('%Y-%m-%d %H:%M:%S', datetime( (time + (21600000 - (time%21600000))) / 1000, 'unixepoch') ) as gap ...
I have two two timestamp fields (START,END) and a TIME_DIFF field which is of Integer type. I am trying to calculate the time between START and END field.. I created a trigger to do that :
CREATE TRIGGER [TIME_DIFF]
AFTER UPDATE OF [END]
ON [KLOG]
BEGIN
update klog set TIME_DIFF =
cast(
(
strftime('%s',KLOG.END) -
strftime('%s',KLOG.START)
) as INT
) / 60/60;
END
This gives me result in whole hours.Anything between 0 and 59 minutes is neglected.
I am wondering how can I modify this trigger so it displays in decimals?
Meaning, if the time difference is 1 hour 59 minutes the result would display 1.59.If the time difference is 35 minutes it would display 0.35.
To interpret a number of seconds as a timestamp, use the unixepoch modifier. Then you can simply use strftime() to format the value:
strftime('%H:%S',
strftime('%s',KLOG.END) - strftime('%s',KLOG.START),
'unixepoch')
If you use Julian days instead of seconds, you do not need a separate modifier:
strftime('%H:%S',
julianday(KLOG.END) - julianday(KLOG.START))