Having an issue where my queries used to work fine with = as in WHERE some_int_field = some_other_int_field. When I do that now I get 0 results. However if I do a WHERE some_int_field LIKE some_other_int_field I get my results. I have checked the length of the fields for hidden characters/spaces and the length of the fields are correct. They are both integer fields. Thoughts? Two tables structure below:
CREATE TABLE "languages"(
"language_id" Integer,
"name" Text,
"english" Text,
"spanish" Text,
"portuguese" Text,
"french" Text );
-- Create index languagesIdx
CREATE INDEX "languagesIdx" ON "languages"( "name" );
BEGIN;
-------------
CREATE TABLE "drop_downs"(
"mode_data" Integer,
"text_index" Integer,
"language_id" Integer );
-- Create index drop_downsIdx
CREATE INDEX "drop_downsIdx" ON "drop_downs"( "mode_data", "language_id" );
BEGIN;
SQLite uses dynamic typing and does not care about the declared type of the fields.
You have strings in your fields.
To check which rows have strings, use something like this:
SELECT * FROM drop_downs WHERE typeof(mode_data) = 'text'
To convert all values in a column into numbers, use something like this:
UPDATE drop_downs SET mode_data = CAST(mode_data AS integer)
Related
When I create index on certain column, index is datatype-aware, however I can store data of any type in any column, but it makes the index somewhat useless if I want to compare column content using different comparison rules, e.g. compare numbers as numbers, not strings.
Example:
sqlite> create table foo (key varchar, value varchar);
sqlite> create index foo_ndx on foo (key, value);
sqlite> insert into foo values ('bar', 10);
sqlite> select * from foo where key = 'bar' and value > 9.0;
sqlite> explain query plan select * from foo where key = 'bar' and value > 9.0;
0|0|0|SEARCH TABLE foo USING COVERING INDEX foo_ndx (key=? AND value>?) (~2 rows)
In this case 9.0 gets implicitly casted as varchar and two varchars get compared, index is used fully, but the record cannot be found.
sqlite> select * from foo where key = 'bar' and CAST(value AS REAL) > 9.0;
bar|10
sqlite> explain query plan select * from foo where key = 'bar'
...> and CAST(value AS REAL) > 9.0;
0|0|0|SEARCH TABLE foo USING COVERING INDEX foo_ndx (key=?) (~5 rows)
In this case I cast value to real explicitly and two reals get compared, sqlite finds the record, but the index used only partially.
This is understandable behaviour, however can I somehow create index on foo table that will treat column value as REAL? I've tried using CAST in CREATE INDEX but got syntax error, I can create view on foo table with CAST, but index cannot be created on view.
Indexes are not data type aware.
By default, values in the value column are treated as text, if possible, because the column has text affinity.
Using an index does not change this behaviour.
If you do not want to treat numbers as text, do not declare the column as varchar. (Note: you can declare columns without any data type.)
I have created a table as below:
CREATE TABLE case_status(data_entry_timestamp DATETIME DEFAULT (datetime('now','localtime')) NOT NULL,
case_number TEXT PRIMARY KEY NOT NULL,
case_name TEXT DEFAULT MISSING,
death_reportdate DATE CONSTRAINT death_reportdate_chk CHECK (death_reportdate==strftime('%Y-%m-%d',death_reportdate)),
);
The column death_reportdate need to have a date with pre-defined format (e.g. 2000-12-31). I created the table, inserted some rows of data, and then try to modified data in death_reportdate, the check rule seems to be bypassed when I enter some random string to it.
What have I done wrong?
You had an extra comma at the end. Correct code:
CREATE TABLE case_status(data_entry_timestamp DATETIME DEFAULT (datetime('now','localtime')) NOT NULL,
case_number TEXT PRIMARY KEY NOT NULL,
case_name TEXT DEFAULT MISSING,
death_reportdate DATE CONSTRAINT death_reportdate_chk CHECK (death_reportdate==strftime('%Y-%m-%d',death_reportdate))
)
it is an old Topic but i had the the same Problem. if the strftime method Fails to Format the string( a bad Input) it retuns null, so you have to check is not null in the end
Here is another solution which works like a charm:
`date` DATE CHECK(date IS strftime('%Y-%m-%d', date))
This also works with the time:
`time` TIME CHECK(time IS strftime('%H:%M:%S', time))
Use this to define your column. I think that is a more elegant solution than checking for null value.
First, two small notes.
I'm using the TEXT type since SQLite does not have "real types." It has 5 column "affinities", INTEGER, TEXT, BLOB, REAL, and NUMERIC. If you say DATE then it uses NUMERIC which can behave a little weirdly in my opinion. I find it best to explicitly use one of the 5 affinities.
I'm using date(...) instead of strftime('%Y-%m-%d', ...) because they are the same thing.
Let's break down why the original question did not work.
DROP TABLE IF EXISTS TEMP.example;
CREATE TEMPORARY TABLE example (
deathdate TEXT CHECK (deathdate == date(deathdate))
);
INSERT INTO TEMP.example (deathdate) VALUES ('2020-01-01');
INSERT INTO TEMP.example (deathdate) VALUES ('a');
INSERT INTO TEMP.example (deathdate) VALUES (NULL);
SELECT * FROM TEMP.example;
Running this lets all three values get into the database. Why? Let's check the documentation for CHECK constraints.
If the result is zero (integer value 0 or real value 0.0), then a constraint violation has occurred. If the CHECK expression evaluates to NULL, or any other non-zero value, it is not a constraint violation.
If you run SELECT 'a' == date('a'); you'll see it is NULL. Why? Check SELECT date('a'); and you'll see it is also NULL. Huh, maybe the documentation for == can help?
Note that there are two variations of the equals and not equals operators. Equals can be either = or ==. [...]
The IS and IS NOT operators work like = and != except when one or both of the operands are NULL. In this case, if both operands are NULL, then the IS operator evaluates to 1 (true) and the IS NOT operator evaluates to 0 (false). If one operand is NULL and the other is not, then the IS operator evaluates to 0 (false) and the IS NOT operator is 1 (true). It is not possible for an IS or IS NOT expression to evaluate to NULL.
We need to use IS, not ==, and trying that we see that 'a' no longer gets in.
DROP TABLE IF EXISTS TEMP.example;
CREATE TEMPORARY TABLE example (
deathdate TEXT CHECK (deathdate IS date(deathdate))
);
INSERT INTO TEMP.example (deathdate) VALUES ('2020-01-01');
INSERT INTO TEMP.example (deathdate) VALUES ('a');
INSERT INTO TEMP.example (deathdate) VALUES (NULL);
SELECT * FROM TEMP.example;
If you don't want NULL to get in, simple change it to deathdate TEXT NOT NULL CHECK (deathdate IS date(deathdate))
This should be an easy one for someone with more SQLite experience than myself.
I need a select statement to get the name out of the following example string:
{"email":"12345678#facebook.com","user_key":"FACEBOOK:12345678","name":"John Smith"}
The output I need is John Smith.
The number of characters before the name is not always the same so a simple substr command won't work. It needs to be dynamic so it can locate where the name starts and then spit it out. I think ltrim or rtrim may help, but even after researching those commands, I don't understand them very well. Also, SQLite doesn't offer instr or position, which might have been helpful, too!
Edit: the schema for this table is as follows:
CREATE TABLE messages (msg_id TEXT PRIMARY KEY, thread_id TEXT, action_id INTEGER, subject TEXT, text TEXT, sender TEXT, timestamp_ms INTEGER, timestamp_sent_ms INTEGER, attachments TEXT, shares TEXT, msg_type INTEGER, affected_users TEXT, coordinates TEXT, offline_threading_id TEXT, source TEXT, channel_source TEXT, is_non_authoritative INTEGER, pending_send_media_attachment STRING, handled_internally_time INTEGER, pending_shares STRING, pending_attachment_fbid STRING, client_tags TEXT, send_error STRING, send_error_message STRING, send_error_timestamp_ms INTEGER, publicity TEXT, tracking TEXT );
CREATE INDEX messages_offline_threading_id_index ON messages ( offline_threading_id );
CREATE INDEX messages_timestamp_index ON messages ( thread_id, timestamp_ms DESC );
CREATE INDEX messages_type_index ON messages ( thread_id, msg_type, timestamp_ms );
The string I have above that I'm working with is from the sender column.
I don't know the language you're using but most of them support user defined functions (http://www.sqlite.org/c3ref/create_function.html) and you could do
select jsonGetName(columnWithJsonText) from messages where ...
It's like a callback to your programming language where you defined which C/Java/PHP function gets called when using jsonGetName() in sqlite.
In that function (in your language of choice) you decode the json string and return the name property.
Ok, so I have a basic table called "ledger", it contains fields of various types, integers, varchar, etc.
In my program, I used to use a query with no "from" predicate to collect all of the rows, which of course works fine. But... I changed my code to allow selecting one row at a time using "where acctno = x" (where X is the account number I want to select at the time).
I thought this must be a bug in the client library for my programming language, so I tested it in the SQLite command-line client - and it still doesn't work!
I am relatively new to SQLite, but I have been using Oracle, MS SQL Server, etc. for years and never seen this type of issue before.
Other things I can tell you:
* Queries using other integer fields also don't work
* Queries on char fields work
* Querying it as a string (with the account number on quotes) still doesn't work. (I thought maybe the numbers were stored as a string inadvertently).
* Accessing rows by rowid works fine - which is why I can edit the database with GUI tools with no noticeable problem.
Examples:
Query with no WHERE (works fine):
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
4|0|0|JPY|14|Japan Post Bank|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
...
Query with WHERE clause: (no results)
sqlite> select * from ledger where acctno=1;
sqlite>
putting quotes around the 1 above changes nothing.
Interestingly enough, "select * from ledger where acctno > 1" returns results! However since it returns ALL results, it's not terrible useful.
I'm sure someone will ask about the table structure, so here goes:
sqlite> .schema ledger
CREATE TABLE "LEDGER" (
"ACCTNO" integer(10,0) NOT NULL,
"drbal" integer(20,0) NOT NULL,
"crbal" integer(20,0) NOT NULL,
"CURRKEY" char(3,0) NOT NULL,
"TEXTKEY" integer(10,0),
"TEXT" VARCHAR(64,0),
"ACCTYPECD" CHAR(1,0) NOT NULL,
"ACCSTCD" CHAR(1,0),
"PACCTNO" number(10,0) NOT NULL,
"CATCD" number(10,0),
"TRANSNO" number(10,0) NOT NULL,
"extrefno" number(10,0),
"UPDATEUSER" VARCHAR(32,0),
"UPDATEDATE" text(8,0),
"UPDATETIME" TEXT(6,0),
"PAYEECD" number(10,0) NOT NULL,
"drbal2" number(10,0) NOT NULL,
"crbal2" number(10,0) NOT NULL,
"delind" boolean,
PRIMARY KEY("ACCTNO"),
CONSTRAINT "fk_curr" FOREIGN KEY ("CURRKEY") REFERENCES "CURRENCY" ("CUR
RKEY") ON DELETE RESTRICT ON UPDATE CASCADE
);
The strangest thing is that I have other similar tables where this works fine!
sqlite> select * from journalhdr where transno=13;
13|Test transaction ATM Withdrawel 20130213|20130223||20130223||
TransNo in that table is also integer (10,0) NOT NULL - this is what makes me thing it is something to do with the values.
Another clue is that the sort order seems to be based on ascii, not numeric:
sqlite> select * from ledger order by acctno;
0|0|0|JPY|11|Root Account|P|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
1|0|0|JPY|8|Paid-In Capital|C|X|0|X|0|0||||0|0|0|
10|0|0|USD|20|Sallie Mae|L|X|0|X|0|0|SYSTEM|20121209|153900|0|0|0|
21|0|0|USD|21|Skrill|A|X|0|X|0|0|SYSTEM|20121209|154000|0|0|0|
22|0|0|USD|22|AES|L|X|0|X|0|0|SYSTEM|20121209|154200|0|0|0|
23|0|0|JPY|23|Marui|L|X|0|X|0|0|SYSTEM|20121209|154400|0|0|0|
24|0|0|JPY|24|Amex JP|L|X|0|X|0|0|SYSTEM|20121209|154500|0|0|0|
3|0|0|JPY|13|Mitsubishi Bank Futsuu|A|X|0|X|0|0|SYSTEM|20121209|150000|0|0|0|
Of course the sort order on journalhdr (where the select works properly) is numeric.
Solved! (sort-of)
The data can be fixed like this:
sqlite> update ledger set acctno = 23 where rowid = 13;
sqlite> select * from ledger where acctno = 25;
25|0|0|JPY|0|Test|L|X|0|X|0|0|SYSTEM|20130224|132500|0|0|0|
Still, if it was stored as strings, then that leave a few questions:
1. Why couldn't I select it as a string using the quotes?
2. How did it get stored as a string since it is a valid integer?
3. How would you go about detecting this problem normally besides noticing bizzarre symptoms?
Although the data would normally be entered by my program, some of it was created by hand using Navicat, so I assume the problem must lie there.
You are victim of SQLite dynamic typing.
Even though SQLite defines system of type affinity, which sets some rules on how input strings or numbers will be converted to actual internal values, but it does NOT prevent software that is using prepared statements to explicitly set any type (and data value) for the column (and this can be different per row!).
This can be shown by this simple example:
CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16));
INSERT INTO ledger VALUES(1, 'John'); -- INTEGER '1'
INSERT INTO ledger VALUES(2 || X'00', 'Zack'); -- BLOB '2\0'
I have inserted second row not as INTEGER, but as binary string containing embedded zero byte. This reproduces your issue exactly, see this SQLFiddle, step by step. You can also execute these commands in sqlite3, you will get the same result.
Below is Perl script that also reproduces this issue
This script creates just 2 rows with acctno having values of integer 1 for first, and "2\0" for second row. "2\0" means string consisting of 2 bytes: first is digit 2, and second is 0 (zero) byte.
Of course, it is very difficult to visually tell "2\0" from just "2", but this is what script below demonstrates:
#!/usr/bin/perl -w
use strict;
use warnings;
use DBI qw(:sql_types);
my $dbh = DBI->connect("dbi:SQLite:test.db") or die DBI::errstr();
$dbh->do("DROP TABLE IF EXISTS ledger");
$dbh->do("CREATE TABLE ledger (acctno INTEGER, name VARCHAR(16))");
my $sth = $dbh->prepare(
"INSERT INTO ledger (acctno, name) VALUES (?, ?)");
$sth->bind_param(1, "1", SQL_INTEGER);
$sth->bind_param(2, "John");
$sth->execute();
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->bind_param(2, "Zack");
$sth->execute();
$sth = $dbh->prepare(
"SELECT count(*) FROM ledger WHERE acctno = ?");
$sth->bind_param(1, "1");
$sth->execute();
my ($num1) = $sth->fetchrow_array();
print "Number of rows matching id '1' is $num1\n";
$sth->bind_param(1, "2");
$sth->execute();
my ($num2) = $sth->fetchrow_array();
print "Number of rows matching id '2' is $num2\n";
$sth->bind_param(1, "2\0", SQL_BLOB);
$sth->execute();
my ($num3) = $sth->fetchrow_array();
print "Number of rows matching id '2<0>' is $num3\n";
Output of this script is:
Number of rows matching id '1' is 1
Number of rows matching id '2' is 0
Number of rows matching id '2<0>' is 1
If you were to look at resultant table using any SQLite tool (including sqlite3), it will print 2 for second row - they all get confused by trailing 0 inside a BLOB when it gets coerced to string or number.
Note that I had to use custom param binding to coerce type to BLOB and permit null bytes stored:
$sth->bind_param(1, "2\0", SQL_BLOB);
Long story short, it is either some of your client programs, or some of client tools like Navicat which screwed it up.
I'm preety new to SQLite.
I have a preety basic question.. Why can't I select rows where specific column equals zero?
The is_unwanted column is type TINYINT (which I see in SQLite basically means INTEGER)
So, I have only one record in the database (for testing).
When I try
SELECT is_unwanted FROM 'urls'
I get a result of "0" (zero), which is fine because that column contains the actual number 0.
I tried =>
SELECT * FROM 'urls' WHERE is_unwanted = 0
And got NO result, but
SELECT * FROM 'urls' WHERE is_unwanted <> 0
gives me result.
What am I doing wrong??
Try running
select '{' || is_unwanted || '}' from urls
to see if the value in the database is really a string containing spaces.
SQLite is a dynamically typed database; when you specify TINYINT is is a hint (SQLite uses the term "affinity") for the column. You can use
select is_unwanted, typeof(is_unwanted) from urls
to see the values with their types.
You could try:
SELECT * FROM urls WHERE coalesce(is_unwanted,'') = ''