I have records containing timestamps, and I want to be able to group by date (according to my local time zone).
The timestamps are already ISO 8601 strings (in the same local time zone with UTC offset specified). I tried using the SQLite date function, however it seems to only give UTC dates:
select date(time), time from mytable;
('2001-07-14', '2001-07-15 09:40:02.500000+10:00'),
('2001-07-14', '2001-07-15 09:39:38.500000+10:00'),
('2001-07-21', '2001-07-22 09:46:09.500000+10:00'),
...
In Postgres I would take a UTC timestamp and do something like ((mystring::timestamp at time zone 'UTC') at time zone 'AEST')::date. I suppose if my timestamps are already expressed with my desired offset, I could just do a string truncation (i.e. substr(time,1,10)), but this seems much less readable code. Can SQLite generally convert timestamps to localised dates?
Can try date(time,'localtime') provided that the target timezone is also currently the system local time zone (which is an awkward requirement).
Otherwise, date(time, '+10 hours').
Related
I have a dataset file with a time variable in "seconds since 1981-01-01 00:00:00". What I need is to convert this time into calendar date (YYYY-MM-DD HH:mm:ss). I've seen a lot of different ways to do this for time since epoch (1970) (timestamp, calendar.timegm, etc) but I'm failing to do this with a different reference date.
One option is to simply add 347133600s (11 years) to each value in seconds. this will then allow you to simply use conversion as it would be from 1970-01-01.
I have a SQLite 3 database on MacOS with timestamp data (typically something like 279020203.539467).
The documentation says that dates can be stored as
REAL - as Julian day numbers, the number of days since noon in Greenwich
on November 24, 4714 B.C. according to the proleptic Gregorian calendar.
And it looks like this is what I've got.
I want to export this and import it into other databases.
I am assuming the timestamp datatype is not compatible between database engines, so some conversion has to happen somewhere, and I am working under the assumption that it'd be best to do that while exporting the data from the SQLite database.
But I can't figure out how to do this conversion.
I've looked at this answer which refers to this forum post which indicates that in SQLite you could do something like
select datetime('40660.9454658044', '+2415018 days', '+12 hours', 'localtime');
(Maybe 2415018 is the number of days between November 24, 4714 B.C. and some other magical date...)
However, replacing the timestamp string in this example with what I have results in null. Presumably because '279020203.539467' is some other kind of timestamp. It is also some magnitudes larger than the example.
But how to convert this to a usable date? I know it should be around 2011/2012.
Interpreting the data as an "integer" (seconds since 1970-01-01) gives 1978 so that is not correct either.
UPDATE: I've found that
288773834.371606 should be 2010-02-25 07.57
296636121.950064 should be 2010-05-27 08.55
(CET if that matters).
The good news is: To convert a Julian date to "regular" date format you could use datetime(strftime('%J',jtime)). FYI Here's the doc for sqlite date and time functions. But there's bad news.
A NASA Calculator computes the Julian date of 2010-02-25 to be 2455246. It computes the civil date of 288773834 as Sept 2, 785907 A.D. sqlite doesn't give that same result using the above notation, but it doesn't give a "date".
Even though the numbers look like Julian date notation, they are not any dates in our lifetimes.
DATEDIF("2010-02-25 07.57"; "2010-05-27 08.55"; "d")
This gives 91 days which works out to be 7862400 seconds, almost exactly the same as the difference between the two timestamps (296636121.950064 - 288773834.371606 = 7862287.578458).
On MacOS native timestamps are seconds from 2001-01-01 (Jan 1 2001).
And
DATEDIF("2001-01-01 00:00"; "2010-02-25 07.57"; "d")
gives 3342 days which is 288748800 seconds, near, but not exactly the timestamp but that difference is caused by
DATEDIF only caters for whole days
CET is one hour off
Correcting for that we get around 25000 seconds more to add, which takes us almost exactly to the timestamp for 2010-02-25 07.57.
So the gist of this is that SQLite on MacOS stores timestamps as MacOS native timestamps which are seconds since start of MacOS epoch (2001-01-01 00.00). This is probably caused by the application that created the data was not using SQLite timestamps but MacOS native dates and storing them in the database as any other data.
Converting this to some other format should be trivial, either during the export, a conversion of the exported file or during imports.
Possibly it would be easiest to convert the date on export from the original database using
select ...
datetime(table.timestamp_field, 'unixepoch', '+31 years')
from ...
I have a dataset with epoch time, but am having difficulty casting it to a timestamp, I also need to parse out the timestamp to add another column just for the time of day, so that I can group data by the time of day the transaction occurred.
I also need to convert to different time zones, from another column (IE, GMT-8 for rows of data, GMT-7 for some, etc etc.)
Example:
1520555554 is March 8th, 2018 16:32 (Pacific Time Zone, GMT-8)
I need to convert from epoch to time stamp, and create another column parsing out 16:32 as the time of day.
For the last part, use strftime() or another one of the sqlite date and time functions. For example,
SELECT strftime('%H:%M', timestamp_column, 'unixepoch') FROM ...
Dealing with timezones is going to be more complicated. Looks like there's no builtin support for them.
I kind of assumed it was a string, so I compared it as a string, but not surprisingly it failed. I believe thats how it works in Mysql. I could be wrong as I haven't worked on it in a while. In either case, how can I check if dates are equal in SQLite? I will be using it in a WHERE clause.
SELECT a._id, b._id, b.start_date,a.event_name, b.start_time,
b.end_date, b.end_time, b.location FROM events_info b INNER JOIN events a ON
a._id=b.event_id WHERE b.start_time = '6:00';
(added space to make it easier to look at)
SQLite doesn't have a dedicated DATETIME type. Normally what people do is make sure they store the date as a formatted string that is consistent; for example, YYYY-MM-DD hh:mm:ss. If you do so, as long as you're consistent, then you can compare dates directly:
SELECT * FROM a WHERE q_date < '2013-01-01 00:00:00';
This works because even though the comparison is technically an alphabetical comparison and not a numeric one, dates in a consistent format like this sort alphabetically as well as numerically.
For such a schema, I would suggest storing dates in 24-hour format (the above example is midnight). Pad months, days, and hours with zeros. If your dates will span multiple timezones, store them all in UTC and do whatever conversion you need client-side to convert them to the local time zone.
Normally dates and times are stored all in one column. If you have to have them separated for whatever reason, just make sure you dates are all consistent and your times are all consistent. For example, dates should all be YYYY-MM-DD and times should all be hh:mm:ss.
The reason that YYYY-MM-DD hh:mm:ss is the preferred format is because when you go from the largest date interval (years) to the smallest (seconds), you can index and sort them very easily and with high performance.
SELECT * FROM a WHERE q_date = '2012-06-04 05:06:00';
would use the index to hone in on the date/time instead of having to do a full table scan. Or if they're in two separate rows:
SELECT * FROM a WHERE q_date = '2012-06-04' AND q_time = '05:06:00';
The key is to make sure that the dates and times are in a consistent format going into the database. For user-friendly presentation, do all conversion client-side, not in the database. (For example, convert '2012-06-04 05:06:00' to "1:06am Eastern 6/4/2012".)
If this doesn't answer question, could you please post the exact format that you're using to store your dates and times, and two example dates that you're trying to compare that aren't working the way you expect them to?
Sqlite can not compare dates directly. we need to convert them in seconds as well as integer also.
Example
SELECT * FROM Table
WHERE
CAST(strftime('%s', date_field) AS integer) <=CAST(strftime('%s', '2015-01-01') AS integer) ;
From Datatypes In SQLite Version 3:
1.2 Date and Time Datatype
SQLite does not have a storage class set aside for storing dates and/or times. Instead, the built-in Date And Time Functions of SQLite are capable of storing dates and times as TEXT, REAL, or INTEGER values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can chose to store dates and times in any of these formats and freely convert between formats using the built-in date and time functions.
If you look at the examples in Date And Time Functions, something like this should get you close to what you want (which, I'm assuming, is 6:00 of the current day):
WHERE b.start_time = date('now', 'start of day', '+6 hours')
I've come into possession of hundreds of ascii data files where the date and time are separate columns like so:
date time
1-Jan-08 23:05
I need to convert this to a usable R Date object, subtract 8 hours (timezone conversion from UTC to Pacific) and then turn it into unix time. I need to do this since the data are collected every evening (from 5pm through 2am the following morning). So if I were to use regular date/time format it would confound days (day1 spans two days when in fact it was just one evening of data collection). I'd like to consider each day's events separately.
Using unixtime will allow me to calculate time differences in events that occur each day (I will probably retain a date field in addition to the unix time). Can someone suggest an efficient way to do this?
Here is some data to use (this is in UTC)
dummy=data.frame(date="1-Jan-08",time="23:05")
Paste them together (which works vectorised) and then parse, e.g.
datetime <- paste(dummy$date, dummy$time)
parsed <- strptime(datetime, "%d-%b-%y %H:%M")
which you can also assign as columns in the data frame.
Edit: strptime() has an optional tz="" argument you can use.