We are using sqlite for our mobile project - however when we try to use our long id variables as primary key autoincrement - the code is giving error "Autoincrement is only allowed in integer primary key". When I checked online about this error, I found out that autoincrement in sqlite is not allowed for bigint types.
The interesting thing is bigint is implemented by integer as well - sqlite has no bigint type - it just uses the size of the integer to decide whether it is a bigint or not.
http://www.sqlite.org/datatype3.html
To fix this - I replaced the open source code where it creates the table with "bigint" with "integer", and wrote some test code to verify that it works outside the boundaries of regular integer (added an item with id larger than the integer range on purpose and continuously added 10 more objects).
It seems like it is working now - yet I want to know if it can cause some other issues. We are syncing the mobile app ids to our db ids, and thus we will definitely have ids larger than the normal integer range.
Is this solution a valid solution? What kind of troubles could this cause?
From Datatypes In SQLite Version:
If the declared type contains the string "INT" then it is assigned INTEGER affinity.
However, this does not mean that a column with INTEGER affinity is suitable for auto-increment duty! The special case is explicitly drawn out in ROWIDs and the INTEGER PRIMARY KEY:
.. A PRIMARY KEY column only becomes an integer primary key if the declared type name is exactly "INTEGER". Other integer type names like "INT" or "BIGINT" or "SHORT INTEGER" or "UNSIGNED INTEGER" causes the primary key column to behave as an ordinary table column with integer affinity and a unique index, not as an alias for the rowid.
The maximum size of an INTEGER value in SQLite is 8 bytes (64bits, signed two's complement) and thus maps cleanly to a long value in .NET (or Java); it doesn't matter if the column was declared as "INTEGER", which it must be for an auto-increment column, or with "BIGINT". (The actual type is per value, not per column; however, auto-increment/ROWID values will all be integers.)
Also, SQLite automatically has a "record/row identifier", even without explicitly creating such a column - this is the "ROWID" column and can be accessed as ROWID, _ROWID_, or OID unless shadowed. See SQLite Autoincrement if this is a suitable option, as it changes the algorithm and removes some monotonically increasing guarantees:
These are important properties in certain applications. But if your application does not need these properties, you should probably stay with the default behavior since the use of AUTOINCREMENT requires additional work..
Related
suppose I have a database like:
CREATE TABLE top_table (
test_id PRIMARY KEY,
cmd TEXT);
CREATE TABLE job_table (
id PRIMARY KEY,
rid integer references top_table(rowid));
If I do a vacuum, would it preserve the relationship "rid integer references top_table(rowid)"? Meaning, would it either leave the top_table rowids unchanged, or would it change them and make corresponding changes to rid in job_table? I want to do the vacuum because I'm merging databases, so I read in a second table, insert its data into an existing table, and drop the second table. If the vacuum won't work properly, my next thought was to create the combined database, drop the table, do a sump and read the sql into a new database. Or is there an easier/cleaner method that I'm unaware of?
Using :-
CREATE TABLE top_table ( test_id PRIMARY KEY, cmd TEXT);
CREATE TABLE job_table ( id PRIMARY KEY, rid integer references top_table(rowid));
Could be an issue as you are referencing the rowid directly rather than referencing an alias of the rowid. Typically an alias of the rowid would be used as per :-
CREATE TABLE top_table ( test_id INTEGER PRIMARY KEY, cmd TEXT);
CREATE TABLE job_table ( id INTEGER PRIMARY KEY, rid INTEGER REFERENCES top_table(test_id));
That is specifying INTEGER PRIMARY KEY, instead of PRIMARY KEY (column affinity NUMERIC) is a special case which makes the column being defined an alias of the rowid, the rowid will then not be subject to change by the VACUUM command as per :-
The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY.
SQLITE -CREATE TABLE - ROWID's and the INTEGER PRIMARY KEY
SQLite - VACUUM
I'm somewhat new at this, but strictly speaking, am I using foreign keys? I'm not declaring it to be a foreign key, and "pragma foreign_keys" returns 0
No BUT only because Foreign KEY enforcement is off (hasn't been turned on), thus the REFERENCES top_table(test_id) has no effect, other than being parsed.
Assuming that you programmatically maintain the referential integrity this should not be an issue.
It can be advantageous to have FOREIGN KEY support on as not only will referential integrity be enforced, the ON UPDATE and ON DELETE actions can then be utilised which can simplify handling updates and deletions by using the CASCADE option (e.g. deleting a parent will a) work without a conflict and b) cascade the deletion so that the children rows will be deleted automatically (if they can be))
SQLite - Enabling Foreign Key Support
SQLite - ON DELETE and ON UPDATE Actions
regarding the comment
test_id is a string, which can be relatively long (60-80 characters or so). the original version of the schema had test_id as a member of both tables. the version that I've shown above (with a bunch of fields removed from both tables for this question) was a check on how much smaller the database got if I switched matching long text strings to matching the rowid which seemed to make more sense than adding a field that serves the same purpose)
In the case where a string is used for referencing will not only save space (in the longer term) BUT the overheads will also be greater than using an alias of the rowid.
In the longer term, because SQLite save chunks (pages). A table will (by default) takes up at least 4k and then 8k ....
First, space wise, not only will the extra 52-80 bytes waste space the parent's column should also be unique. If UNIQUE were specified then there is the additional overhead of an index.
However, with an alias of the rowid, then 8 bytes max for the integer, which is used anyway unless the table is defined using WITHOUT ROWID. If I recall there is an extra byte for the flag that the column is an alias of the rowid. The rowid and therefore alias has to be unique and the index (as such) exists. Furthermore due to it being central to SQLite accessing by rowid can be twice as fast.
In short an alias of the rowid is probably the best option in many (probably most) cases for relationships.
I am looking at migrating a small sqlite3 db to mysql. I know mysql but new to sqlite3 so have been reading about it online. I used pragma table_info(<table_name>) to get info about the table structure.
From the output I could understand columns with data type TEXT, INTEGER but i do not understand datatype BINARY(32). From sqlite3 documentation on the net there is a BINARY collation, but there is no BINARY datatype. So I just want to understand this this BINARY(32) datatype. Thanks.
SQLite is unusual in datatypes (column types). You can store any type of data in any type of columns with the exception of the rowid column or an alias of the rowid column.
see Rowid Tables
rowid is similar to MySQL AUTO INCREMENT BUT beware of differences
In the example below see how the rowid starts from -100, then -99 .....
AUTOINCREMENT on SQLite is only a constraint as such that enforces that a new id is higher than any existing in the table.
So BINARY, BINARY(32), (rumplestistkin even) are valid for the datatype when defining a column.
However, a column will be given a column affinity and governed by the rules :-
If the column type contains INT the the affinity is INTEGER.
If the column type contains CHAR, CLOB or TEXT, then it's affinity is TEXT.
If the column type contains BLOB then it's affinity is BLOB.
If the column type contains REAL FLOA or DOUB then it's affinity is REAL.
Otherwise the affinity is NUMERIC.
As such BINARY(32) is NUMERIC affinity. However, the column type is of little consequence in regards to storing data. The affinity can affect retrieval a little.
In regard to converting the rules mentioned above could be utilised you could also perhaps find the typeof function of use (example of it's use is in the example along with the results). However, neither will necessarily, indicate how the data is subsequently used which could well be a factor that needs consideration.
SQLite's flexibility with column types aids in converting from other relational databases BUT can be a bit of a hindrance when converting from SQLite.
Note this answer is by no means intended to be comprehensive explanation of the conversion from SQLite to MysQL.
See Datatypes in SQLite
Here's an example that shows that any type can be stored in any column (thus any row/col combination can store different types) :-
DROP TABLE IF EXISTS example;
CREATE TABLE IF NOT EXISTS example (
rowid_alias_must_be_unique_integer INTEGER PRIMARY KEY, -- INTEGER PRIMARY KEY makes the column an alias of the rowid
col_text TEXT,
col_integer INTEGER,
col_real REAL,
col_BLOB BLOB,
col_anyother this_is_a_stupid_column_type
);
INSERT INTO example VALUES (-100,'MY TEXT', 340000,34.5678,x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',100);
INSERT INTO example (col_text,col_integer,col_real,col_blob,col_anyother) VALUES
('MY TEXT','MY TEXT','MY TEXT','MY TEXT','MY TEXT'),
(100,100,100,100,100),
(34.5678,34.5678,34.5678,34.5678,34.5678),
(x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff',x'f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff')
;
SELECT
*,
rowid,
typeof(rowid_alias_must_be_unique_integer),
typeof(col_text),
typeof(col_integer),
typeof(col_real),
typeof(col_blob),
typeof(col_anyother)
FROM example
;
DROP TABLE IF EXISTS example;
Running the above results in (Note different SQLtools handle blobs in different ways, Navicat was used to run the above) :-
note that the typeof function returns the storage type as opposed to the affinity. However, the affinity can affect the storage type.
e.g. if the affinity is text then with the exception of a blob the value is stored as text. (see 2. in Datatype in SQLite above).
After reading https://sqlite.org/datatype3.html which states
"SQLite does not have a storage class set aside for storing dates
and/or times."
but able to run this
CREATE TABLE User (ID INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, BORN_ON DATE NULL)
and then see it in "DB Browser for SQL" like this:
I start to wonder if SQLite does support Date type of it is just "faking" the support using other types. And even if so why the DB Browser see it as a Date? Any meta info stored inside the DB?
SQLite does not fake Date with Numerics.
There is no Date data type in SQLite.
In Datatypes In SQLite Version 3 it is explained clearly that:
SQLite uses a more general dynamic type system
Instead of data types there are 5 Storage Classes: NULL, INTEGER, REAL, TEXT and BLOB.
Also:
Any column in an SQLite version 3 database, except an INTEGER PRIMARY
KEY column, may be used to store a value of any storage class.
So when you use Date as the data type of a column in the CREATE TABLE statement you are not restricted to store in it only date-like values. Actually you can store anything in that column.
Tools like "DB Browser for SQLite" and others may offer various data types to select from to define a column when you create the table.
The selection of the data type that you make is not restrictive, but it is rather indicative of what type of data you want to store in a column.
In fact, you can create a table without even declaring the data types of the columns:
CREATE TABLE tablename(col1, col2)
or use fictional data types:
CREATE TABLE tablename(col1 somedatatype, col2 otherdatatype)
and insert values of any data type:
INSERT INTO tablename(col1, col2) VALUES
(1, 'abc'),
('XYZ', '2021-01-06'),
(null, 3.5)
Based on what Colonel Thirty Two suggested (read more on the page) it seems that when you declare a field as Date its affinity will be numeric.
So SQLite "fakes" Date with Numerics.
And even if so why the DB Browser see it as a Date? Any meta info stored inside the DB?
Yes, it simply stores the type name used when the column was created. The linked page calls it "declared type". In this case you get NUMERIC affinity (DATE is even given as one of the examples in 3.1.1) and it behaves like any other column with this affinity:
A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT. For the purposes of this paragraph, hexadecimal integer literals are not considered well-formed and are stored as TEXT. (This is done for historical compatibility with versions of SQLite prior to version 3.8.6 2014-08-15 where hexadecimal integer literals were first introduced into SQLite.) If a floating point value that can be represented exactly as an integer is inserted into a column with NUMERIC affinity, the value is converted into an integer. No attempt is made to convert NULL or BLOB values.
A string might look like a floating-point literal with a decimal point and/or exponent notation but as long as the value can be expressed as an integer, the NUMERIC affinity will convert it into an integer. Hence, the string '3.0e+5' is stored in a column with NUMERIC affinity as the integer 300000, not as the floating point value 300000.0.
So if you insert dates looking like e.g. "2021-01-05" they will be stored as strings. But
you can also insert strings which don't look like dates.
if you insert "20210105" it will be stored as the number 20210105.
You can use CHECK constraints to prevent inserting non-date strings.
See also https://sqlite.org/lang_datefunc.html which says what (string and number) formats date/time functions expect.
I was psyched about the possibility of using SQLite as a database solution during development so that I could focus on writing the code first and dynamically generating the db at runtime using NHibernate's ShemaExport functionality. However, I'm running into a few issues, not the least of which is that it seems that SQLite requires me to use Int64 for my primary keys (vs, say, Int32 or Guid). Is there any way around this?
Note: I should specify that this is in the context of an app using NHibernate. It is not strictly speaking the case that one can't create a table in SQLite with an INT datatype, but the behavior when you save and retrieve the data seems to indicate that it's being stored and/or retrieved as Int64.
SQLite will let you use any field in your table as a PRIMARY KEY. Doing so will implicitly create a UNIQUE index on the field. This is then the field that you, as a developer, can consider to be the primary unique identifier for the field. It can be any supported SQLite data type (below).
SQLite will always create an implicit internal numeric identifier for every table. It will have several aliases including RowID, OID, and _ROWID_. If you create your primary key as INTEGER PRIMARY KEY then it will use the same field as your primary key and SQLite's internal numeric identifier.
SQLite doesn't have a concept of Int32 or Int64 or Guid data types. It only has four data types: INT, REAL, TEXT, and BLOB. When you run DDL against SQLite if you use anything other than these four identifiers, SQLite will use a set of rules to determine which type to use. Basically, Int32 and Int64 are treated as aliases of INT and end up doing the exact same thing.
Even once you've created the tables with the data types you mentioned for each field, all you set is the type affinity for that field. SQLite does not enforce data types. Any data can be put into any field regardless of the declared type. SQLite will use the type affinity to convert data if possible, so if you insert '123' as a text string into an INT field, it will store it as the number 123.
The only exception to the type affinity is INTEGER PRIMARY KEY FIELDS. Those must be integers.
Integers in SQLite are always stored with a variable length field. So depending on the size of the integer, you may actually get an Int32 back for some rows an Int64 for others, all within the same field. This depends on the wrapper you're using, in this case NHibernate (I guess with System.Data.SQLite).
It does not require you to use Int64, however, it is possible that it only allows that when you specify a numeric primary key. Because sqlite doesn't really have referential integrity checking (though there has been recent discussion of this and perhaps dr hipp has even implemented, i haven't checked lately), all primary key means is "Make this column unique and create an index on it". there isn't much special about it. You can certainly use varchar or text for a primary key. for example, this works:
create table t_test (
theID varchar(36) primary key,
nm varchar(50)
)
in the above you could use theID to store a guid in text form.
More info can be found here: http://www.sqlite.org/lang_createtable.html#rowid
#weenet ... per your comments, the following code works just fine.
i think you need to post your code if you're still having troubles.
create table t_test2 (
theID int32 primary key,
nm varchar(50)
);
insert into t_test2 (theID, nm) values (1, 'don');
insert into t_test2 (theID, nm) values (2, 'weenet');
select * from t_test2;
additionally, this code works fine (varchar as a primary key):
create table t_test (
theID varchar(36) primary key,
nm varchar(50)
)
insert into t_test (theID, nm) values ('abcdefg', 'don');
insert into t_test (theID, nm) values ('hijklmnop', 'weenet');
select * from t_test
When creating a table in SQLite3, I get confused when confronted with all the possible datatypes which imply similar contents, so could anyone tell me the difference between the following data-types?
INT, INTEGER, SMALLINT, TINYINT
DEC, DECIMAL
LONGCHAR, LONGVARCHAR
DATETIME, SMALLDATETIME
Is there some documentation somewhere which lists the min./max. capacities of the various data-types? For example, I guess smallint holds a larger maximum value than tinyint, but a smaller value than integer, but I have no idea of what these capacities are.
SQLite, technically, has no data types, there are storage classes in a manifest typing system, and yeah, it's confusing if you're used to traditional RDBMSes. Everything, internally, is stored as text. Data types are coerced/converted into various storage locations based on affinities (ala data types assigned to columns).
The best thing that I'd recommend you do is to :
Temporarily forget everything you used to know about standalone database datatypes
Read the above link from the SQLite site.
Take the types based off of your old schema, and see what they'd map to in SQLite
Migrate all the data to the SQLite database.
Note: The datatype limitations can be cumbersome, especially if you add time durations, or dates, or things of that nature in SQL. SQLite has very few built-in functions for that sort of thing. However, SQLite does provide an easy way for you to make your own built-in functions for adding time durations and things of that nature, through the sqlite3_create_function library function. You would use that facility in place of traditional stored procedures.
The difference is syntactic sugar. Only a few substrings of the type names matter as for as the type affinity is concerned.
INT, INTEGER, SMALLINT, TINYINT → INTEGER affinity, because they all contain "INT".
LONGCHAR, LONGVARCHAR → TEXT affinity, because they contain "CHAR".
DEC, DECIMAL, DATETIME, SMALLDATETIME → NUMERIC, because they don't contain any of the substrings that matter.
The rules for determining affinity are listed at the SQLite site.
If you insist on strict typing, you can implement it with CHECK constraints:
CREATE TABLE T (
N INTEGER CHECK(TYPEOF(N) = 'integer'),
Str TEXT CHECK(TYPEOF(Str) = 'text'),
Dt DATETIME CHECK(JULIANDAY(Dt) IS NOT NULL)
);
But I never bother with it.
As for the capacity of each type:
INTEGER is always signed 64-bit. Note that SQLite optimizes the storage of small integers behind-the-scenes, so TINYINT wouldn't be useful anyway.
REAL is always 64-bit (double).
TEXT and BLOB have a maximum size determined by a preprocessor macro, which defaults to 1,000,000,000 bytes.
Most of those are there for compatibility. You really only have integer, float, text, and blob. Dates can be stored as either a number (unix time is integer, microsoft time is float) or as text.
NULL. The value is a NULL value.
INTEGER. The value is a signed integer, stored in 1, 2, 3, 4, 6, or 8 bytes depending on the magnitude of the value.
REAL. The value is a floating point value, stored as an 8-byte IEEE floating point number.
TEXT. The value is a text string, stored using the database encoding (UTF-8, UTF-16BE or UTF-16LE).
BLOB. The value is a blob of data, stored exactly as it was input.
As an addition to answer from dan04, if you want to blindly insert a NUMERIC other than zero represented by a TEXT but ensure that text is convertible to a numeric:
your_numeric_col NUMERIC CHECK(abs(your_numeric_col) <> 0)
Typical use case is in a query from a program that treats all data as text (for uniformity & simplicity, since SQLite already does so). The nice thing about this is that it allows constructs like this:
INSERT INTO table (..., your_numeric_column, ...) VALUES (..., some_string, ...)
which is convenient in case you're using placeholders because you don't have to handle such non-zero numeric fields specially. An example using Python's sqlite3 module would be,
conn_or_cursor.execute(
"INSERT INTO table VALUES (" + ",".join("?" * num_values) + ")",
str_value_tuple) # no need to convert some from str to int/float
In the above example, all values in str_value_tuple will be escaped and quoted as strings when passed to SQlite. However, since we're not checking explicitly the type via TYPEOF but only convertibility to type, it will still work as desired (i.e., SQLite will either store it as a numeric or fail otherwise).