I'm trying to decide what is the best way to store a datetime in sqlite. The date will be in epoch.
I've been reading on wiki about the 2038 problem (it's very much like the year 2000 problem). Taking this into account with what I've been reading on tutorialspoint:
From https://www.tutorialspoint.com/sqlite/sqlite_data_types.htm
Tutorialspoint suggests using the below data types form datetime.
SQLite does not have a separate storage class for storing dates and/or times, but SQLite is capable of storing dates and times as TEXT, REAL or INTEGER values.
But when I looked at the type descriptions, BLOB didn't have a size limit and represents the data as it is inserted into the database.
BLOB The value is a blob of data, stored exactly as it was input.
INTEGER The value is a signed integer, stored in 1, 2, 3, 4, 6, or 8 bytes depending on the magnitude of the value.
I saw on tutorials point that they suggest using sqlite type INTEGER for datetime. But taken with 2038 problem, I'm thinking that using BLOB is a better choice if I'm focusing on future proofing because BLOB does not have a dependence on a specific number of bytes like INTEGER does depend.
I'm new to database design, so I'm wondering what's best to do?
INTEGER as it says can be up to 8 bytes i.e. a 64 bit signed integer. Your issue is not SQLite being able to store values not subject to the 2038 issue with 32 bits. Your issue will be in retrieving a time from something that is not subject to the issue, that is unless you are trying to protect against the year 292,277,026,596 problem.
There is no need to use a BLOB and the added complexity and additional processing of converting between a BLOB and the time.
It may even be that you can use SQLite itself to retrieve suitable values, if you wanted to store the current time or a time based upon the current time aka now.
Perhaps consider the following :-
DROP TABLE IF EXISTS timevalues;
/* Create the table with 1 column with a weird type and a default value as now (seconds since Jan 1st 1970)*/
CREATE TABLE IF NOT EXISTS timevalues (dt typedoesnotmatterthtamuch DEFAULT (strftime('%s','now')));
/* INSERT 2 rows with dates of 1000 years from now */
INSERT INTO timevalues VALUES
(strftime('%s','now','+1000 years')),
((julianday('now','+1000 years') - 2440587.5)*86400.0);
/* INSERT a row using the DEFAULT */
INSERT INTO timevalues (rowid) /* specify the rowid column so there is no need to supply value for the dt column */
VALUES ((SELECT count() FROM timevalues)+1 /* get the highest rowid + 1 */);
/* Retrieve the data rowid, the value as stored in the dt column and the dt column converted to a user friendly format */
SELECT rowid,*, datetime(dt,'unixepoch') AS userfriendly FROM timevalues;
/* Cleanup the Environment */
DROP TABLE IF EXISTS timevalues;
Which results in :-
You would probably want to have a read of Date And Time Functions e.g. for strftime, julianday and now
rowid is a special normally hidden column that exists for all table unless it is WITHOUT ROWID table. It wouldn't typically be used, or if so aliased by using INTEGER PRIMARY KEY
see SQLite Autoincrement to find out about rowid and alias thereof and why not to use AUTOINCREMENT.
a column type of typedoesnotmatterthtamuch see Datatypes In SQLite Version 3 as to why this can be.
Related
I am looking at migrating a small sqlite3 db to mysql. I know mysql but new to sqlite3 so have been reading about it online. I used pragma table_info(<table_name>) to get info about the table structure.
From the output I could understand columns with data type TEXT, INTEGER but i do not understand datatype BINARY(32). From sqlite3 documentation on the net there is a BINARY collation, but there is no BINARY datatype. So I just want to understand this this BINARY(32) datatype. Thanks.
SQLite is unusual in datatypes (column types). You can store any type of data in any type of columns with the exception of the rowid column or an alias of the rowid column.
see Rowid Tables
rowid is similar to MySQL AUTO INCREMENT BUT beware of differences
In the example below see how the rowid starts from -100, then -99 .....
AUTOINCREMENT on SQLite is only a constraint as such that enforces that a new id is higher than any existing in the table.
So BINARY, BINARY(32), (rumplestistkin even) are valid for the datatype when defining a column.
However, a column will be given a column affinity and governed by the rules :-
If the column type contains INT the the affinity is INTEGER.
If the column type contains CHAR, CLOB or TEXT, then it's affinity is TEXT.
If the column type contains BLOB then it's affinity is BLOB.
If the column type contains REAL FLOA or DOUB then it's affinity is REAL.
Otherwise the affinity is NUMERIC.
As such BINARY(32) is NUMERIC affinity. However, the column type is of little consequence in regards to storing data. The affinity can affect retrieval a little.
In regard to converting the rules mentioned above could be utilised you could also perhaps find the typeof function of use (example of it's use is in the example along with the results). However, neither will necessarily, indicate how the data is subsequently used which could well be a factor that needs consideration.
SQLite's flexibility with column types aids in converting from other relational databases BUT can be a bit of a hindrance when converting from SQLite.
Note this answer is by no means intended to be comprehensive explanation of the conversion from SQLite to MysQL.
See Datatypes in SQLite
Here's an example that shows that any type can be stored in any column (thus any row/col combination can store different types) :-
DROP TABLE IF EXISTS example;
CREATE TABLE IF NOT EXISTS example (
rowid_alias_must_be_unique_integer INTEGER PRIMARY KEY, -- INTEGER PRIMARY KEY makes the column an alias of the rowid
col_text TEXT,
col_integer INTEGER,
col_real REAL,
col_BLOB BLOB,
col_anyother this_is_a_stupid_column_type
);
INSERT INTO example VALUES (-100,'MY TEXT', 340000,34.5678,x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',100);
INSERT INTO example (col_text,col_integer,col_real,col_blob,col_anyother) VALUES
('MY TEXT','MY TEXT','MY TEXT','MY TEXT','MY TEXT'),
(100,100,100,100,100),
(34.5678,34.5678,34.5678,34.5678,34.5678),
(x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff')
;
SELECT
*,
rowid,
typeof(rowid_alias_must_be_unique_integer),
typeof(col_text),
typeof(col_integer),
typeof(col_real),
typeof(col_blob),
typeof(col_anyother)
FROM example
;
DROP TABLE IF EXISTS example;
Running the above results in (Note different SQLtools handle blobs in different ways, Navicat was used to run the above) :-
note that the typeof function returns the storage type as opposed to the affinity. However, the affinity can affect the storage type.
e.g. if the affinity is text then with the exception of a blob the value is stored as text. (see 2. in Datatype in SQLite above).
After reading https://sqlite.org/datatype3.html which states
"SQLite does not have a storage class set aside for storing dates
and/or times."
but able to run this
CREATE TABLE User (ID INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, BORN_ON DATE NULL)
and then see it in "DB Browser for SQL" like this:
I start to wonder if SQLite does support Date type of it is just "faking" the support using other types. And even if so why the DB Browser see it as a Date? Any meta info stored inside the DB?
SQLite does not fake Date with Numerics.
There is no Date data type in SQLite.
In Datatypes In SQLite Version 3 it is explained clearly that:
SQLite uses a more general dynamic type system
Instead of data types there are 5 Storage Classes: NULL, INTEGER, REAL, TEXT and BLOB.
Also:
Any column in an SQLite version 3 database, except an INTEGER PRIMARY
KEY column, may be used to store a value of any storage class.
So when you use Date as the data type of a column in the CREATE TABLE statement you are not restricted to store in it only date-like values. Actually you can store anything in that column.
Tools like "DB Browser for SQLite" and others may offer various data types to select from to define a column when you create the table.
The selection of the data type that you make is not restrictive, but it is rather indicative of what type of data you want to store in a column.
In fact, you can create a table without even declaring the data types of the columns:
CREATE TABLE tablename(col1, col2)
or use fictional data types:
CREATE TABLE tablename(col1 somedatatype, col2 otherdatatype)
and insert values of any data type:
INSERT INTO tablename(col1, col2) VALUES
(1, 'abc'),
('XYZ', '2021-01-06'),
(null, 3.5)
Based on what Colonel Thirty Two suggested (read more on the page) it seems that when you declare a field as Date its affinity will be numeric.
So SQLite "fakes" Date with Numerics.
And even if so why the DB Browser see it as a Date? Any meta info stored inside the DB?
Yes, it simply stores the type name used when the column was created. The linked page calls it "declared type". In this case you get NUMERIC affinity (DATE is even given as one of the examples in 3.1.1) and it behaves like any other column with this affinity:
A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT. For the purposes of this paragraph, hexadecimal integer literals are not considered well-formed and are stored as TEXT. (This is done for historical compatibility with versions of SQLite prior to version 3.8.6 2014-08-15 where hexadecimal integer literals were first introduced into SQLite.) If a floating point value that can be represented exactly as an integer is inserted into a column with NUMERIC affinity, the value is converted into an integer. No attempt is made to convert NULL or BLOB values.
A string might look like a floating-point literal with a decimal point and/or exponent notation but as long as the value can be expressed as an integer, the NUMERIC affinity will convert it into an integer. Hence, the string '3.0e+5' is stored in a column with NUMERIC affinity as the integer 300000, not as the floating point value 300000.0.
So if you insert dates looking like e.g. "2021-01-05" they will be stored as strings. But
you can also insert strings which don't look like dates.
if you insert "20210105" it will be stored as the number 20210105.
You can use CHECK constraints to prevent inserting non-date strings.
See also https://sqlite.org/lang_datefunc.html which says what (string and number) formats date/time functions expect.
Given we have a simple table like
CREATE TABLE A(
amount INTEGER
);
What is the difference between queries
INSERT INTO A VALUES(4);
and
INSERT INTO A VALUES('12');
As seen in schema, amount is an INTEGER column. The first query operates with just that - an integer, but the second one operates with a string '12'. Yet both queries work just fine, the table gets values 4 and 12, and can select or, say, sum them up correctly as two valid Integers:
SELECT sum(amount) AS "Total" FROM A;
correctly yields 16.
So is there a difference between inserting an integer as (4) and inserting it as ('12') into the INTEGER-type column?
SQLite tries to convert your String into an Integer before inserting the value into your table as described in the manual.
The type affinity of a column is the recommended type for data stored in that column. The important idea here is that the type is recommended, not required. Any column can still store any type of data. It is just that some columns, given the choice, will prefer to use one storage class over another.
Taken straight off of SQLite's site "The value is a signed integer, stored in 1, 2, 3, 4, 6, or 8 bytes depending on the magnitude of the value."
Does this mean that if you have 1 value that requires 8 bytes, ALL values in that column will be treated as 8 bytes. Or, if the rest are all 1 byte, and one value is 8 bytes, will only that value be using 8 bytes and the rest will remain at 1?
I'm more used to SQL in which you specify the integer size accordingly.
I know the question seems trivial, but based on the answer will determine how I handle a piece of the database.
The sqlite database structure is different in the way it handles data types. Each field can have a different type...
Here is the documentation from sqlite:
Most SQL database engines use static typing. A datatype is associated with each column
in a table and only values of that particular datatype are allowed to be stored in that
column. SQLite relaxes this restriction by using manifest typing. In manifest typing, the
datatype is a property of the value itself, not of the column in which the value is
stored. SQLite thus allows the user to store any value of any datatype into any column
regardless of the declared type of that column. (There are some exceptions to this rule:
An INTEGER PRIMARY KEY column may only store integers. And SQLite attempts to coerce
values into the declared datatype of the column when it can.)
I have a SQlite3 table that has typeless columns like in this example:
CREATE TABLE foo(
Timestamp INT NOT NULL,
SensorID,
Value,
PRIMARY KEY(Timestamp, SensorID)
);
I have specific reasons not to declare the type of the columns SensorID and Value.
When inserting rows with numeric SensorID and Value columns I notice that they are being written as plain text into the .db file.
When I change the CREATE TABLE statement to...
CREATE TABLE foo(
Timestamp INT NOT NULL,
SensorID INT,
Value REAL,
PRIMARY KEY(Timestamp, SensorID)
);
...then the values seem to be written in some binary format to the .db file.
Since I need to write several millions of rows to the database, I have concerns about the file size this data produces and so would like to avoid value storage in plain text form.
Can I force SQLite to use binary representation in it's database file without using explicitly typed columns?
Note: Rows are currently written with PHP::PDO using prepared statements.
The example in section 3.4 in the sqlite docs about types demonstrates the insertion of a number as int in a column without an explicit declaration of type. I guess the trick is leaving out the quotes around the number, which would convert it to a string (which, in the case of typed columns, would be coerced back into a number).
Section 2 in the page linked above also provides a lot of info about the type conversions taking place.