I pull data from a websource and it returns time in the format 'YYYY-MM-DDTHH:MM:SS.SSSZ'. Since I would be doing a lot of comparisons on this column, I thought it would be best to store them as the number of millliseconds since epoch (I need millisecond precision!!). What would be the best way to convert the incoming datetime string to number of milliseconds?
I tried to use
strftime('%s', datetimeString) - gives back the number of seconds since epoch till datetimeString
strftime('%f', datetimeString) - gives back only the SS.SSS part!!
I'm guessing it has something to do with SQLite not having actual Date or Datetime types (section 1.2). Instead it formats them as one of:
an ISO8601 string (TEXT type)
a floating point number of days (REAL type)
an integer number of seconds since the epoch time (INTEGER type)
The REAL data type may not have enough precision to store a date, time, and fractional seconds in a single field.
It seems the answer is one of:
store them as two separate fields, the datetime in one, and the fractional seconds in another
do the integer number of milliseconds calculation yourself and store the integer result in a single column, so long as the maximum level of fractional second precision you need will fit in a signed integer of 8-bytes or fewer (the maximum size for the INTEGER type in SQLite3)
switch databases to a DBMS that supports real date/datetime types
Without switching your DB, and if you never need sub-millisecond precision, then I'd at least try the "manual calculation + single integer column" solution and see if it works.
Related
The sqlite timestamp here is 18 digits, but how can one know its specific date. 132079170460000000 should correspond to July 2019, but this year cannot be obtained regardless of interpretation or other conversions. I want to get time like 2019-07-30 13:23:40.
This is sqllite timestamp, but product specific model does not know,only know that this timestamp is 18-digit just like these:
132079170420000000
132079218060000000
If 132089746940000000 is equivalent to "2019-07-31 13:42:31" as mentioned in the comments, then try this:
SELECT datetime( DateColumnWith18Digits / 1.0E11 + 1137798.60179213 )
FROM SomeTable
If the example you shared was just a random value, this technique might still work, but you'll need to find the offset factor. The conversion assumes that the 18-digit value is the number of milliseconds since some epoch. I simply assumed that one of the comments contained a valid equivalence and found the offset to match the Julian date value.
The reverse encoding would be like
SELECT (julianday('2019-07-31 13:42:31') - 1137798.60179213) * 1E11
So, I have a following SQLite query:
SELECT datetime('2013-10-07 08:23:19.111') = datetime('2013-10-07 08:23:19.222')
To my surprise the result that I get is ... 1 (true). No matter what values I put after the dot, it is always true when fractional values are different.
According to docs:
In formats 4, 7, and 10, the fractional seconds value SS.SSS can have one or more digits following the decimal point. Exactly three digits are shown in the examples because only the first three digits are significant to the result, but the input string can have fewer or more than three digits and the date/time functions will still operate correctly.
Is there some sort of mistake or is there something I don't understand?
The page you linked to says:
The datetime() function returns "YYYY-MM-DD HH:MM:SS".
If you know that your values already are exactly in the correct format, just compare the strings directly. Otherwise, replace datetime() with strftime() with %f to convert the values into the format you want.
I was asked to create a query to pull a near-real-time report from an Informix database (I have select access only, I cannot create a SP) and I felt like I succeeded pretty well until I realized that there was a discrepancy in a datetime field. it seems that the program that is populating the db is hard-coded to enter the time in the datetime field in UTC (five hours off of the local time. When the time was 2:30 it entered a row in the database saying John Doe completed the task at 7:30). In my report I am supposed to calculate the number of seconds (as an int) since the user completed the task (field is "completionTime") and I was originally just using:
sysdate - completionTime interval seconds(9) to seconds cast to char then cast to int
When I realized the mistake in the timezone of the completionTime field I just subtracted the offset as an integer (I was already converting the interval to an integer, so I just adjusted the answer by 18000). This worked just fine until Daylight Saving started. Then all of a sudden local time was 4 hours (14400 seconds instead of 18000) off of UTC.
Since I can only select from the db, I next tried using an inefficient case statement (my query went from <0.5 seconds to 3-5 seconds for only 25 rows). Following a suggestion from another forum I changed the time to an integer of seconds from the unix epoch, then used the dbinfo('utc_to_datetime') sp to convert it back to a datetime in the right timezone.
This approach works, but the calculation looks terrible to me:
cast(cast(cast((sysdate - dbinfo("utc_to_datetime", cast(cast(cast((completionTime - TO_DATE('Friday January 1, 2010 0:00', '%A %B %d, %Y %R')) as interval second(9) to second) as char(10)) as int) +1262304000)) as interval second(9) to second) as char(10)) as int)
notice that I am calculating the length of time from the completiontime to 1-1-2010 then adding 12 billion seconds (going all the way back to the unix epoch is too big for Informix's interval seconds(9) to second, hence the two-steps) so that I can then plug it into the dbinfo("utc_to_datetime") sp to convert it back to a datetime in the right timezone, then subtracting it from sysdate. The worst part (besides the six casts) is that the completiontimes that I am dealing with are all within 24 hours of sysdate, most are within 10 minutes, yet I am adding on 12 billion seconds so that I can use the only function I can find that converts between timezones.
My question is, Is this really the best way to do this? By the way, this works very quickly, and my query is back down to a reasonable execution time (<0.5 seconds), I'm just looking at this query and thinking that there has got to be a better way.
Jared
Maybe instead of sysdate you can use DBINFO('utc_current'):
SELECT DBINFO('utc_current') - (completionTime interval seconds(9) to seconds) FROM ...
Below is my table create in SQLite database,
CREATE TABLE MyData(
Code VARCHAR(20),
Amount DECIMAL(18, 8)
);
then I insert 2 rows into the table.
INSERT INTO MyData
VALUES('A', 1.12345678);
INSERT INTO MyData
VALUES('B', 1234567890.12345678);
After that, execute a SELECT statement,
SELECT * FROM MyData;
SQLite returns the following result:
A|1.12345678
B|1234567890.12346
The DECIMAL(18, 8) suppose means precision=18 and scale=8, why some decimal places are truncated?
The details of how sqlite stores its data is described here. When you specify the DECIMAL column type, the storage for the column has NUMERIC affinity.
Section 2.0 has the following description about type affinity:
A column with NUMERIC affinity may contain values using all five
storage classes. When text data is inserted into a NUMERIC column, the
storage class of the text is converted to INTEGER or REAL (in order of
preference) if such conversion is lossless and reversible. For
conversions between TEXT and REAL storage classes, SQLite considers
the conversion to be lossless and reversible if the first 15
significant decimal digits of the number are preserved. If the
lossless conversion of TEXT to INTEGER or REAL is not possible then
the value is stored using the TEXT storage class. No attempt is made
to convert NULL or BLOB values.
This indicates that sqlite will attempt conversions between types, and if the first 15 digits of the number can be converted and reversed, the numbers are deemed to be equal. This effectively puts a limit on the available precision with which a number can be stored to 15 significant digits.
The wikipedia article on double precision floating point numbers has additional information which is useful when dealing with floating point numbers.
I kind of assumed it was a string, so I compared it as a string, but not surprisingly it failed. I believe thats how it works in Mysql. I could be wrong as I haven't worked on it in a while. In either case, how can I check if dates are equal in SQLite? I will be using it in a WHERE clause.
SELECT a._id, b._id, b.start_date,a.event_name, b.start_time,
b.end_date, b.end_time, b.location FROM events_info b INNER JOIN events a ON
a._id=b.event_id WHERE b.start_time = '6:00';
(added space to make it easier to look at)
SQLite doesn't have a dedicated DATETIME type. Normally what people do is make sure they store the date as a formatted string that is consistent; for example, YYYY-MM-DD hh:mm:ss. If you do so, as long as you're consistent, then you can compare dates directly:
SELECT * FROM a WHERE q_date < '2013-01-01 00:00:00';
This works because even though the comparison is technically an alphabetical comparison and not a numeric one, dates in a consistent format like this sort alphabetically as well as numerically.
For such a schema, I would suggest storing dates in 24-hour format (the above example is midnight). Pad months, days, and hours with zeros. If your dates will span multiple timezones, store them all in UTC and do whatever conversion you need client-side to convert them to the local time zone.
Normally dates and times are stored all in one column. If you have to have them separated for whatever reason, just make sure you dates are all consistent and your times are all consistent. For example, dates should all be YYYY-MM-DD and times should all be hh:mm:ss.
The reason that YYYY-MM-DD hh:mm:ss is the preferred format is because when you go from the largest date interval (years) to the smallest (seconds), you can index and sort them very easily and with high performance.
SELECT * FROM a WHERE q_date = '2012-06-04 05:06:00';
would use the index to hone in on the date/time instead of having to do a full table scan. Or if they're in two separate rows:
SELECT * FROM a WHERE q_date = '2012-06-04' AND q_time = '05:06:00';
The key is to make sure that the dates and times are in a consistent format going into the database. For user-friendly presentation, do all conversion client-side, not in the database. (For example, convert '2012-06-04 05:06:00' to "1:06am Eastern 6/4/2012".)
If this doesn't answer question, could you please post the exact format that you're using to store your dates and times, and two example dates that you're trying to compare that aren't working the way you expect them to?
Sqlite can not compare dates directly. we need to convert them in seconds as well as integer also.
Example
SELECT * FROM Table
WHERE
CAST(strftime('%s', date_field) AS integer) <=CAST(strftime('%s', '2015-01-01') AS integer) ;
From Datatypes In SQLite Version 3:
1.2 Date and Time Datatype
SQLite does not have a storage class set aside for storing dates and/or times. Instead, the built-in Date And Time Functions of SQLite are capable of storing dates and times as TEXT, REAL, or INTEGER values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can chose to store dates and times in any of these formats and freely convert between formats using the built-in date and time functions.
If you look at the examples in Date And Time Functions, something like this should get you close to what you want (which, I'm assuming, is 6:00 of the current day):
WHERE b.start_time = date('now', 'start of day', '+6 hours')