It's my first time with sqlite and I wrote a simple database for storing and retrieving map tiles.
Here is the sqlite data base:
CREATE TABLE IF NOT EXISTS Tiles (id INTEGER NOT NULL PRIMARY KEY, X INTEGER NOT NULL, Y INTEGER NOT NULL, Zoom INTEGER NOT NULL, Type INTEGER NOT NULL,Date TEXT)
CREATE TABLE IF NOT EXISTS TilesData (id INTEGER NOT NULL PRIMARY KEY CONSTRAINT fk_Tiles_id REFERENCES Tiles(id) ON DELETE CASCADE, Tile BLOB NULL)
CREATE TRIGGER fki_TilesData_id_Tiles_id BEFORE INSERT ON [TilesData] FOR EACH ROW BEGIN SELECT RAISE(ROLLBACK, 'insert on table TilesData violates foreign key constraint fki_TilesData_id_Tiles_id') WHERE (SELECT id FROM Tiles WHERE id = NEW.id) IS NULL; END
CREATE TRIGGER fku_TilesData_id_Tiles_id BEFORE UPDATE ON [TilesData] FOR EACH ROW BEGIN SELECT RAISE(ROLLBACK, 'update on table TilesData violates foreign key constraint fku_TilesData_id_Tiles_id') WHERE (SELECT id FROM Tiles WHERE id = NEW.id) IS NULL; END
CREATE TRIGGER fkdc_TilesData_id_Tiles_id BEFORE DELETE ON Tiles FOR EACH ROW BEGIN DELETE FROM TilesData WHERE TilesData.id = OLD.id; END
Here is the insert operation:
INSERT INTO Tiles(X, Y, Zoom, Type,Date) VALUES(?, ?, ?, ?,?);
INSERT INTO TilesData(id, Tile) VALUES((SELECT last_insert_rowid()), ?);
Here is the read operation:
SELECT Tile FROM TilesData WHERE id = (SELECT id FROM Tiles WHERE X=%1 AND Y=%2 AND Zoom=%3 AND Type=%4)
The problem is, when the database is small the read is still very fast(around 20ms), but when the database becomes larger (around 15000 rows), the read access becomes very slow (around 4000ms).
Am I doing anything wrong? Any suggestions to improve the performance?
if you want for Android You can use also noSQL solution for speed. For example Couchbase
Related
In a sqlite3 database I would like to create a trigger on a view so that I can insert data over the view. Inside the trigger I would like to insert something in the tblmedia table. The id of the inserted row should be now also inserted into the tblbook as id.
In sqlite there are no variables. Otherwise I would store the returning value in the variable and would use it in the second query.
Can this even be achieved in sqlite?
Following my sql schema:
CREATE TABLE tblmedia(
id INTEGER PRIMARY KEY NOT NULL,
title VARCHAR NOT NULL,
raiting INTEGER,
file_name VARCHAR NOT NULL,
media_type TEXT NOT NULL
);
CREATE TABLE tblbook(
id INTEGER PRIMARY KEY NOT NULL,
author VARCHAR,
FOREIGN KEY (id) REFERENCES tblmedia(id) ON DELETE CASCADE
);
CREATE VIEW book AS
SELECT
m.id as id,
m.title as title,
b.author as author,
m.raiting as raiting,
m.file_name as file_name
FROM tblbook b
LEFT JOIN tblmedia m ON m.id = b.id;
CREATE TRIGGER insert_book
INSTEAD OF INSERT ON book
BEGIN
INSERT INTO tblmedia(title, raiting, file_name)
VALUES(new.title, new.raiting, new.file_name);
INSERT INTO tblbook(id, author)
VALUES (xx, new.author); -- xx should be the id from the previous insert
END
I have a table of phone numbers (tblPhoneNumbers):
ID, UserID, PhoneNumber
and I need to move them into a Users table (tblUsers) that contains:
ID, PhoneNumber1, PhoneNumber2
tblPhoneNumbers is assumed to have 2 rows for every user. Is it possible to move the PhoneNumber value of the first row into PhoneNumber1, and the PhoneNumber value of the second row into PhoneNumber2?
Essentially this is reverse-normalization but this is the task I need help with.
Thanks!
I need to use SQLite so I cannot use any syntax not available to SQLite.
If you're using sqlite 3.25 or better, you can use window functions to do it all in one statement (I assume here that the UserID column from tblPhoneNumbers is a foreign key that references ID from tblUsers, and that the given userid already has a record in that table; adjust as needed):
WITH allnumbers AS
(SELECT UserID
, PhoneNumber
, row_number() OVER (PARTITION BY UserID) AS num
FROM tblPhoneNumbers)
UPDATE tblUsers AS t
SET PhoneNumber1 = (SELECT a.PhoneNumber
FROM allnumbers AS a
WHERE a.UserID = t.ID AND num = 1)
, PhoneNumber2 = (SELECT a.PhoneNumber
FROM allnumbers AS a
WHERE a.UserID = t.ID AND num = 2);
(And if your system only has an older version that don't support window functions, you can always download a copy of the latest version of the sqlite3 shell and use it instead of the OS provided one).
(edit: You'll want an index on tblPhoneNumbers.UserID for better performance)
You could use the following :-
-- Create a temporary swap table
CREATE TEMP TABLE IF NOT EXISTS swapPhoneNumbers (ID INTEGER PRIMARY KEY, UserID INTEGER, PhoneNumber TEXT, replacementPhoneNumber TEXT);
-- Clear the temporary swap table in case it's used more than once
DELETE FROM swapPhoneNumbers;
-- Populate the temporary swap table according to the original data
INSERT INTO swapPhoneNumbers (ID,UserID,PhoneNumber) SELECT * FROM tblPhoneNumbers;
-- Update the swap table to include the replacement phone numbers
UPDATE swapPhoneNumbers SET replacementPhoneNumber = (
SELECT PhoneNumber FROM tblPhoneNumbers
WHERE swapPhoneNumbers.userID = tblPhoneNumbers.userID
AND swapPhoneNumbers.ID <> tblPhoneNumbers.ID
);
-- Update the original table with the new phone numbers
UPDATE tblPhoneNumbers SET PhoneNumber = (
SELECT replacementPhoneNumber FROM swapPhoneNumbers
WHERE tblPhoneNumbers.ID = swapPhoneNumbers.ID
);
The following is the SQL used to test the above.
-- Create Testing Table with some data
DROP TABLE IF EXISTS tblphoneNumbers;
CREATE TABLE IF NOT EXISTS tblPhoneNumbers (ID INTEGER PRIMARY KEY, userID INTEGER, PhoneNumber TEXT);
INSERT INTO tblPhoneNumbers (userID, PhoneNumber) VALUES
(1,'0111111111'),(1,'0222222222'),(2,'0333333333'),(2,'0444444444'),(3,'0555555555'),(3,'0666666666')
;
-- Show what is in the original table
SELECT * FROM tblPhoneNumbers;
-- Create a temporary swap table
CREATE TEMP TABLE IF NOT EXISTS swapPhoneNumbers (ID INTEGER PRIMARY KEY, UserID INTEGER, PhoneNumber TEXT, replacementPhoneNumber TEXT);
-- Clear the temporary swap table in case it's used more than once
DELETE FROM swapPhoneNumbers;
-- Populate the temporary swap table according to the original data
INSERT INTO swapPhoneNumbers (ID,UserID,PhoneNumber) SELECT * FROM tblPhoneNumbers;
-- Show what is in the swap table
SELECT * FROM swapPhoneNumbers;
-- Update the swap table to include the replacement phone numbers
UPDATE swapPhoneNumbers SET replacementPhoneNumber = (
SELECT PhoneNumber FROM tblPhoneNumbers
WHERE swapPhoneNumbers.userID = tblPhoneNumbers.userID
AND swapPhoneNumbers.ID <> tblPhoneNumbers.ID
);
-- Show what is now in the swap table
SELECT * FROM swapPhoneNumbers;
-- Update the original table with the new phone numbers
UPDATE tblPhoneNumbers SET PhoneNumber = (
SELECT replacementPhoneNumber FROM swapPhoneNumbers
WHERE tblPhoneNumbers.ID = swapPhoneNumbers.ID
);
-- Show what is in the original table
SELECT * FROM tblPhoneNumbers;
And this is some screen shots from doing it
Good day SO,
I'm working on a program in PowerShell to manipulate an SQLite DB I created. I've never written a serious applications to utilize a DB so right now I'm super interested in optimizing my querys, so I'm really interested in feed back. My primary issue is I have a lot of data that I want to include in a separate table that may or not exist already. All my research really seemed to lead to perform an INSERT and let the UNIQUE constraints sort it out, than do a select on the new record which seemed like two table scans and inefficient. So my solution was Create a temp table, insert into the temp table FROM the table with data I want and perform an INSERT if the data was not in the temporary table. I'm a few drinks in tonight and haven't tested the code so please don't critique small typos, I just want to know if my methodology is out to lunch, and if so please provide better direction.
My table is as shown:
CREATE TABLE Processes (
pk INTEGER PRIMARY KEY AUTOINCREMENT
UNIQUE,
hostname INTEGER NOT NULL,
artifacttype INTEGER REFERENCES ArtifactType (pk),
processname INTEGER REFERENCES ProcessesName (pk),
filelocation INTEGER NOT NULL
REFERENCES files (pk),
pid INTEGER,
ppid INTEGER,
starttime INTEGER,
stoptime INTEGER,
token STRING,
logonid INTEGER,
exitstatus INTEGER,
threadcount INTEGER,
commandline INTEGER REFERENCES ProcessesCommandline (pk),
user INTEGER REFERENCES users (pk),
PeakVirtualSize INTEGER,
VirtualSize INTEGER,
PeakWorkingSetSize INTEGER,
suspicious BOOLEAN,
malicious BOOLEAN
);
Transaction:
#"
CREATE TEMPORARY TABLE IF NOT EXISTS Results(pk INTEGER, data TEXT);
INSERT INTO Results(pk, data) VALUES ((SELECT pk, name FROM ProcessesName WHERE name = #processname));
INSERT INTO ProcessesName(name) VALUES (SELECT #processname WHERE NOT EXISTS (SELECT pk FROM Results WHERE data = #processname));
INSERT INTO Results (pk, data) VALUES ((SELECT last_insert_row_id, #processname WHERE NOT EXISTS (SELECT pk FROM Results WHERE data =#processname)));
INSERT INTO Results(pk, data) VALUES ((SELECT pk, file FROM Files WHERE file = #filelocation));
INSERT INTO Files(file) VALUES (SELECT #filelocation WHERE NOT EXISTS (SELECT pk FROM Results WHERE data = #filelocation));
INSERT INTO Results (pk, data) VALUES ((SELECT last_insert_row_id, #filelocation WHERE NOT EXISTS (SELECT pk FROM Results WHERE data =#filelocation)));
INSERT INTO Results(pk, data) VALUES ((SELECT pk, commandline FROM ProcessesCommandline WHERE commandline = #commandline));
INSERT INTO ProcesseCommandline(commandline) VALUES (SELECT #commandline WHERE NOT EXISTS (SELECT pk FROM Results WHERE data = #commandline));
INSERT INTO Results (pk, data) VALUES ((SELECT last_insert_row_id, #filelocation WHERE NOT EXISTS (SELECT pk FROM Results WHERE data =#commandline)));
INSERT INTO Results(pk, data) VALUES ((SELECT pk, SID FROM Users WHERE SID = #SID));
INSERT INTO Users(SID) VALUES (SELECT #SID WHERE NOT EXISTS (SELECT pk FROM Results WHERE data = #filelocation));
INSERT INTO Results (pk, data) VALUES ((SELECT last_insert_row_id, #SID WHERE NOT EXISTS (SELECT pk FROM Results WHERE data =#SID)));
INSERT INTO processes(hostname, artifacttype, processname, filelocation, pid, ppid, starttime, threadcount, commandline, user, PeakVirtualSize, VirtualSize, PeakWorkingSetSize)
VALUES (#hostname, #artifacttype, (SELECT pk FROM Results WHERE data = #processname), (SELECT pk FROM Results WHERE data #filelocation), #pid, #ppid, #starttime, #threadcount, (SELECT pk FROM Results where data = #commandline), SELECT pk FROM Results WHERE data = #SID, #PeakVirtualSize, #VirtualSize, #PeakWorkingSetSize);
DROP TABLE Results;
"#
*there are a few foreign keys where the data is being tracked application side so no complex queries are required.
So my core question is, is there a more efficient way to do this?
Thanks guys!
I am new to database and have a sqlite database like this:
CREATE TABLE IF NOT EXISTS Tiles (id INTEGER NOT NULL PRIMARY KEY, X INTEGER NOT NULL, Y INTEGER NOT NULL, Zoom INTEGER NOT NULL, Type INTEGER NOT NULL,Date TEXT)
CREATE TABLE IF NOT EXISTS TilesData (id INTEGER NOT NULL PRIMARY KEY CONSTRAINT fk_Tiles_id REFERENCES Tiles(id) ON DELETE CASCADE, Tile BLOB NULL)
If I call the following insert call for the first time:
INSERT INTO Tiles(X, Y, Zoom, Type,Date) VALUES(?, ?, ?, ?,?)
INSERT INTO TilesData(id, Tile) VALUES((SELECT last_insert_rowid()), ?)
What would be the id of Tiles and TilesData?
What's the function of SELECT last_insert_rowid() here?
I built your queries on both SQLFiddle (to easily show you the results) and in a local SQLite instance (to confirm that SQLite (SQL.js) behaves the same as a local instance) using the following script:
CREATE TABLE IF NOT EXISTS Tiles (id INTEGER NOT NULL PRIMARY KEY, X INTEGER NOT NULL, Y INTEGER NOT NULL, Zoom INTEGER NOT NULL, Type INTEGER NOT NULL,Date TEXT);
CREATE TABLE IF NOT EXISTS TilesData (id INTEGER NOT NULL PRIMARY KEY CONSTRAINT fk_Tiles_id REFERENCES Tiles(id) ON DELETE CASCADE, Tile BLOB NULL);
INSERT INTO Tiles(X, Y, Zoom, Type,Date) VALUES(1, 2, 3, 4, 'Date');
INSERT INTO TilesData(id, Tile) VALUES((SELECT last_insert_rowid()), 'Stuff');
select * from Tiles;
select * from TilesData;
In both instances, the id in both tables is 1.
The purpose of select last_insert_rowid() is to determine what the rowid is of the last successful insert. In your example, it is returning the value from the INSERT INTO Tiles query (1) and then assigning that to the id of TilesData.
We can demonstrate this works by also specifying the id in your first insert and seeing that the id of TilesData also matches the forced id.
INSERT INTO Tiles(id, X, Y, Zoom, Type,Date) VALUES(123456, 10, 2, 3, 4, 'Date');
INSERT INTO TilesData(id, Tile) VALUES((SELECT last_insert_rowid()), 'Stuff');
In this example, I'm forcing the id in the Tiles table to be 123456. This value is utilized as the id for the TilesData.
select * from TilesData;
Results in:
id | Tile
--------------
123456 | Stuff
I'm using SQLite to play around and learn some more SQL. I have a SQLite 3 database populated like this:
create table playlist (id integer primary key autoincrement, name text);
create table playlistitem (id integer primary key autoincrement,
playlist_id integer, name text);
insert into playlist (name) values ("Moss");
insert into playlist (name) values ("Jen");
insert into playlistitem (playlist_id, name) values (1, "Roy");
insert into playlistitem (playlist_id, name) values (1, "Richmond");
insert into playlistitem (playlist_id, name) values (2, "Denholm");
Great, now I have two playlist items in the "Moss" playlist, "Roy" and "Richmond"; I have one item in the "Jen" playlist: "Denholm".
What I'd like to do is delete the "Moss" playlist and all of its items with a single query.
I saw something like this, which fails for me:
delete playlist, playlistitem from playlist
inner join playlistitem on playlistitem.playlist_id = playlist.id
where playlist.name = "Moss";
Failure:
Error: near "playlist": syntax error
What am I doing wrong?
sqlite doesn't support join in delete statement. You have to use separate query that deletes from second table based on playlist_id, making a delete trigger on playlist, or make that reference a foreign key with on delete cascade:
create table playlistitem (
id integer primary key autoincrement,
playlist_id integer, name text,
foreign key(playlist_id) references playlist(id) on delete cascade);
and then just using delete from playlist where name='Moss'.
Don't forget to enable foreign keys - pragma foreign_keys=1 (you have to re-enable this on each sqlite connection, e.g. as the first command after connecting).