sqlite3 binary data type - sqlite

I am looking at migrating a small sqlite3 db to mysql. I know mysql but new to sqlite3 so have been reading about it online. I used pragma table_info(<table_name>) to get info about the table structure.
From the output I could understand columns with data type TEXT, INTEGER but i do not understand datatype BINARY(32). From sqlite3 documentation on the net there is a BINARY collation, but there is no BINARY datatype. So I just want to understand this this BINARY(32) datatype. Thanks.

SQLite is unusual in datatypes (column types). You can store any type of data in any type of columns with the exception of the rowid column or an alias of the rowid column.
see Rowid Tables
rowid is similar to MySQL AUTO INCREMENT BUT beware of differences
In the example below see how the rowid starts from -100, then -99 .....
AUTOINCREMENT on SQLite is only a constraint as such that enforces that a new id is higher than any existing in the table.
So BINARY, BINARY(32), (rumplestistkin even) are valid for the datatype when defining a column.
However, a column will be given a column affinity and governed by the rules :-
If the column type contains INT the the affinity is INTEGER.
If the column type contains CHAR, CLOB or TEXT, then it's affinity is TEXT.
If the column type contains BLOB then it's affinity is BLOB.
If the column type contains REAL FLOA or DOUB then it's affinity is REAL.
Otherwise the affinity is NUMERIC.
As such BINARY(32) is NUMERIC affinity. However, the column type is of little consequence in regards to storing data. The affinity can affect retrieval a little.
In regard to converting the rules mentioned above could be utilised you could also perhaps find the typeof function of use (example of it's use is in the example along with the results). However, neither will necessarily, indicate how the data is subsequently used which could well be a factor that needs consideration.
SQLite's flexibility with column types aids in converting from other relational databases BUT can be a bit of a hindrance when converting from SQLite.
Note this answer is by no means intended to be comprehensive explanation of the conversion from SQLite to MysQL.
See Datatypes in SQLite
Here's an example that shows that any type can be stored in any column (thus any row/col combination can store different types) :-
DROP TABLE IF EXISTS example;
CREATE TABLE IF NOT EXISTS example (
rowid_alias_must_be_unique_integer INTEGER PRIMARY KEY, -- INTEGER PRIMARY KEY makes the column an alias of the rowid
col_text TEXT,
col_integer INTEGER,
col_real REAL,
col_BLOB BLOB,
col_anyother this_is_a_stupid_column_type
);
INSERT INTO example VALUES (-100,'MY TEXT', 340000,34.5678,x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',100);
INSERT INTO example (col_text,col_integer,col_real,col_blob,col_anyother) VALUES
('MY TEXT','MY TEXT','MY TEXT','MY TEXT','MY TEXT'),
(100,100,100,100,100),
(34.5678,34.5678,34.5678,34.5678,34.5678),
(x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff')
;
SELECT
*,
rowid,
typeof(rowid_alias_must_be_unique_integer),
typeof(col_text),
typeof(col_integer),
typeof(col_real),
typeof(col_blob),
typeof(col_anyother)
FROM example
;
DROP TABLE IF EXISTS example;
Running the above results in (Note different SQLtools handle blobs in different ways, Navicat was used to run the above) :-
note that the typeof function returns the storage type as opposed to the affinity. However, the affinity can affect the storage type.
e.g. if the affinity is text then with the exception of a blob the value is stored as text. (see 2. in Datatype in SQLite above).

Related

SQLITE get generated column type

After creating an sqlite table with a generated column in it, the type only shows up, if it was specified, and there can be cases when junk gets into the type description as well.
Example:
create table test(
id integer primary key not null,
gen generated always as (id * 2) stored
);
Using pragma table_xinfo(test); afterwards nets the following output:
0|id|INTEGER|1||1|0
1|gen||0||0|3
The type is simply missing from the correct column.
If the column were to be created with:
gen integer generated always as (id * 2) stored
instead, then the type would correctly show up as INTEGER.
Are there any methods that would get the type of a column in a table without having to resort to parsing the table creation code?
Nevermind, as usual, I find the answer right after asking it. According to sqlite documentation:
The datatype and collating sequence of the generated column are determined only by the datatype and COLLATE clause on the column definition. The datatype and collating sequence of the GENERATED ALWAYS AS expression have no affect on the datatype and collating sequence of the column itself.
Which I assume means, that just as in other places, if the datatype is not specified, it is thought of as a blob.

Does SQLite actually support DATE type?

After reading https://sqlite.org/datatype3.html which states
"SQLite does not have a storage class set aside for storing dates
and/or times."
but able to run this
CREATE TABLE User (ID INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, BORN_ON DATE NULL)
and then see it in "DB Browser for SQL" like this:
I start to wonder if SQLite does support Date type of it is just "faking" the support using other types. And even if so why the DB Browser see it as a Date? Any meta info stored inside the DB?
SQLite does not fake Date with Numerics.
There is no Date data type in SQLite.
In Datatypes In SQLite Version 3 it is explained clearly that:
SQLite uses a more general dynamic type system
Instead of data types there are 5 Storage Classes: NULL, INTEGER, REAL, TEXT and BLOB.
Also:
Any column in an SQLite version 3 database, except an INTEGER PRIMARY
KEY column, may be used to store a value of any storage class.
So when you use Date as the data type of a column in the CREATE TABLE statement you are not restricted to store in it only date-like values. Actually you can store anything in that column.
Tools like "DB Browser for SQLite" and others may offer various data types to select from to define a column when you create the table.
The selection of the data type that you make is not restrictive, but it is rather indicative of what type of data you want to store in a column.
In fact, you can create a table without even declaring the data types of the columns:
CREATE TABLE tablename(col1, col2)
or use fictional data types:
CREATE TABLE tablename(col1 somedatatype, col2 otherdatatype)
and insert values of any data type:
INSERT INTO tablename(col1, col2) VALUES
(1, 'abc'),
('XYZ', '2021-01-06'),
(null, 3.5)
Based on what Colonel Thirty Two suggested (read more on the page) it seems that when you declare a field as Date its affinity will be numeric.
So SQLite "fakes" Date with Numerics.
And even if so why the DB Browser see it as a Date? Any meta info stored inside the DB?
Yes, it simply stores the type name used when the column was created. The linked page calls it "declared type". In this case you get NUMERIC affinity (DATE is even given as one of the examples in 3.1.1) and it behaves like any other column with this affinity:
A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT. For the purposes of this paragraph, hexadecimal integer literals are not considered well-formed and are stored as TEXT. (This is done for historical compatibility with versions of SQLite prior to version 3.8.6 2014-08-15 where hexadecimal integer literals were first introduced into SQLite.) If a floating point value that can be represented exactly as an integer is inserted into a column with NUMERIC affinity, the value is converted into an integer. No attempt is made to convert NULL or BLOB values.
A string might look like a floating-point literal with a decimal point and/or exponent notation but as long as the value can be expressed as an integer, the NUMERIC affinity will convert it into an integer. Hence, the string '3.0e+5' is stored in a column with NUMERIC affinity as the integer 300000, not as the floating point value 300000.0.
So if you insert dates looking like e.g. "2021-01-05" they will be stored as strings. But
you can also insert strings which don't look like dates.
if you insert "20210105" it will be stored as the number 20210105.
You can use CHECK constraints to prevent inserting non-date strings.
See also https://sqlite.org/lang_datefunc.html which says what (string and number) formats date/time functions expect.

SQLITE3 store text and numbers in the same column

In the SQLITE3 Docu you can find:
"SQLite uses a more general dynamic type system. In SQLite, the datatype of a value is associated with the value itself, not with its container. The dynamic type system of SQLite is backwards compatible with the more common static type systems of other database engines in the sense that SQL statements that work on statically typed databases should work the same way in SQLite. However, the dynamic typing in SQLite allows it to do things which are not possible in traditional rigidly typed databases."
I have a column with different values some are integer, some float and some text. I use BLOB als storage class and it works fine. But i'm a little suspicious of this solution. Is this the right way to store numbers and text in the same column?
TLDR; you are not doing it wrong, but I also think you are not doing what you think you are doing.
I don't think you are using a BLOB storage class, but a BLOB type affinity. In SQLite there are 2 concepts of datatype. Storage class, and type affinity.
Storage class is the way a given value is stored in memory, it describes how bit patterns represent values. E.g. while numerically equal, the integer 1 and the floating point 1.0 are represented by very different bit patterns (at least in the memory, but curiously "As an internal optimization, small floating point values with no fractional component and stored in columns with REAL affinity are written to disk as integers").
There are 5 storage classes, NULL, INTEGER, REAL, TEXT, and BLOB.
Type affinity on the other hand, is not a value- but a column-level concept. Type affinity is a characteristic of a given column. There are also 5 type affinities, NUMBER, INTEGER, REAL, TEXT, and BLOB.
The first thing you might notice, is that there is no 1-1 mapping between storage classes and affinities. That's because they are indeed almost completely unrelated. Consider the following
sqlite> CREATE TABLE Test(A TEXT, B BLOB, C REAL);
sqlite> INSERT INTO Test VALUES ('text', 'also a text', 'another one');
sqlite> INSERT INTO Test VALUES (10, 10.1, 10.11111111111111111111111);
sqlite> SELECT A, TYPEOF(A), B, TYPEOF(B), C, TYPEOF(C) FROM Test;
text|text|also a text|text|another one|text
10|text|10.1|real|10.1111111111111|real
As you can see, I defined the columns with types TEXT, BLOB, and REAL. Those are the affinities. But I was able to insert values of different types into them, regardless of their affinities. E.g. column C now both stores 'another one', of storage class TEXT, and 10.1111111111111 of storage class REAL.
Normally you should not be too concerned about storage classes, as you can see from my example, SQLite just automatically and correctly inferred them from the data I inserted.
Type affinity is more important. That's what describes how your data is compared, sorted, how it's uniqueness is determined. E.g.
sqlite> CREATE Table T(A TEXT);
sqlite> INSERT INTO T VALUES ('1'), (2), (0.0);
sqlite> SELECT * FROM T ORDER BY A ASC;
0.0
1
2
sqlite> CREATE TABLE R(A REAL);
sqlite> INSERT INTO R VALUES ('1'), (2), (0.0);
sqlite> SELECT * FROM R ORDER BY A ASC;
0.0
1.0
2.0
sqlite> CREATE TABLE B(A BLOB);
sqlite> INSERT INTO B VALUES ('1'), (2), (0.0);
sqlite> SELECT * FROM B ORDER BY A ASC;
0.0
2
1
So for practical purposes, it is more closely related to your intuitive idea of a datatype. Think about what your data means, and choose the affinity accordingly. If you store your values in a column of BLOB affinity, a bitwise comparison will be used on them. If this is what you want, go for it. But if you want a lexicographic comparison you need TEXT affinity, or if you want to do the comparisons based on the numerical values your data represents, then e.g. REAL.

sqlite typeless columns and storage in .db file

I have a SQlite3 table that has typeless columns like in this example:
CREATE TABLE foo(
Timestamp INT NOT NULL,
SensorID,
Value,
PRIMARY KEY(Timestamp, SensorID)
);
I have specific reasons not to declare the type of the columns SensorID and Value.
When inserting rows with numeric SensorID and Value columns I notice that they are being written as plain text into the .db file.
When I change the CREATE TABLE statement to...
CREATE TABLE foo(
Timestamp INT NOT NULL,
SensorID INT,
Value REAL,
PRIMARY KEY(Timestamp, SensorID)
);
...then the values seem to be written in some binary format to the .db file.
Since I need to write several millions of rows to the database, I have concerns about the file size this data produces and so would like to avoid value storage in plain text form.
Can I force SQLite to use binary representation in it's database file without using explicitly typed columns?
Note: Rows are currently written with PHP::PDO using prepared statements.
The example in section 3.4 in the sqlite docs about types demonstrates the insertion of a number as int in a column without an explicit declaration of type. I guess the trick is leaving out the quotes around the number, which would convert it to a string (which, in the case of typed columns, would be coerced back into a number).
Section 2 in the page linked above also provides a lot of info about the type conversions taking place.

Can I use anything other than BIGINT as Primary Key data type in SQLite?

I was psyched about the possibility of using SQLite as a database solution during development so that I could focus on writing the code first and dynamically generating the db at runtime using NHibernate's ShemaExport functionality. However, I'm running into a few issues, not the least of which is that it seems that SQLite requires me to use Int64 for my primary keys (vs, say, Int32 or Guid). Is there any way around this?
Note: I should specify that this is in the context of an app using NHibernate. It is not strictly speaking the case that one can't create a table in SQLite with an INT datatype, but the behavior when you save and retrieve the data seems to indicate that it's being stored and/or retrieved as Int64.
SQLite will let you use any field in your table as a PRIMARY KEY. Doing so will implicitly create a UNIQUE index on the field. This is then the field that you, as a developer, can consider to be the primary unique identifier for the field. It can be any supported SQLite data type (below).
SQLite will always create an implicit internal numeric identifier for every table. It will have several aliases including RowID, OID, and _ROWID_. If you create your primary key as INTEGER PRIMARY KEY then it will use the same field as your primary key and SQLite's internal numeric identifier.
SQLite doesn't have a concept of Int32 or Int64 or Guid data types. It only has four data types: INT, REAL, TEXT, and BLOB. When you run DDL against SQLite if you use anything other than these four identifiers, SQLite will use a set of rules to determine which type to use. Basically, Int32 and Int64 are treated as aliases of INT and end up doing the exact same thing.
Even once you've created the tables with the data types you mentioned for each field, all you set is the type affinity for that field. SQLite does not enforce data types. Any data can be put into any field regardless of the declared type. SQLite will use the type affinity to convert data if possible, so if you insert '123' as a text string into an INT field, it will store it as the number 123.
The only exception to the type affinity is INTEGER PRIMARY KEY FIELDS. Those must be integers.
Integers in SQLite are always stored with a variable length field. So depending on the size of the integer, you may actually get an Int32 back for some rows an Int64 for others, all within the same field. This depends on the wrapper you're using, in this case NHibernate (I guess with System.Data.SQLite).
It does not require you to use Int64, however, it is possible that it only allows that when you specify a numeric primary key. Because sqlite doesn't really have referential integrity checking (though there has been recent discussion of this and perhaps dr hipp has even implemented, i haven't checked lately), all primary key means is "Make this column unique and create an index on it". there isn't much special about it. You can certainly use varchar or text for a primary key. for example, this works:
create table t_test (
theID varchar(36) primary key,
nm varchar(50)
)
in the above you could use theID to store a guid in text form.
More info can be found here: http://www.sqlite.org/lang_createtable.html#rowid
#weenet ... per your comments, the following code works just fine.
i think you need to post your code if you're still having troubles.
create table t_test2 (
theID int32 primary key,
nm varchar(50)
);
insert into t_test2 (theID, nm) values (1, 'don');
insert into t_test2 (theID, nm) values (2, 'weenet');
select * from t_test2;
additionally, this code works fine (varchar as a primary key):
create table t_test (
theID varchar(36) primary key,
nm varchar(50)
)
insert into t_test (theID, nm) values ('abcdefg', 'don');
insert into t_test (theID, nm) values ('hijklmnop', 'weenet');
select * from t_test

Resources