I have a python script which creates a sqlite database out of some external data. This works fine. But everytime I execute a GROUP BY query on this database, I get an "Error: unable to open database file". Normal SELECT queries work.
This is an issue for both, the sqlite3 library of python and the sqlite3 cli binary:
sqlite> SELECT count(*) FROM REC;
count(*)
----------
528489
sqlite> SELECT count(*) FROM REC GROUP BY VERSION;
Error: unable to open database file
sqlite>
I know that these errors are typically permission errors (I have read all questions which I could find on this topic on StackOverflow), but I'm quite sure it is not in my case:
I'm talking about a readily created database and read requests
I checked the permissions: Both the file and its containing folders have write permissions set
I can even write to the database: Creating a new table is no problem.
The device is not full, it got plenty of space.
Ensure that your process has access to the TEMP directory.
From the SQLite's Use Of Temporary Disk Files documentation:
SQLite may make use of transient indices to implement SQL language
features such as:
An ORDER BY or GROUP BY clause
The DISTINCT keyword in an aggregate query
Compound SELECT statements joined by UNION, EXCEPT, or INTERSECT
Each transient index is stored in its own temporary file. The
temporary file for a transient index is automatically deleted at the
end of the statement that uses it.
You probably can verify if temporary storage is the problem by setting the temp_store pragma to MEMORY:
PRAGMA temp_store = MEMORY;
to tell SQLite to keep the transient index for the GROUP BY clause in memory.
Alternatively, create an explicit index on the column you are grouping by to prevent the transient index from being created.
Related
I open the database
sqlite3 /nix/var/nix/db/db.sqlite
in the database, I try to list every table and then select the content of a table
.tables
Error: attempt to write a readonly database
select * from t_I_dont_know_if_this_table_exists;
Parse error: attempt to write a readonly database
This database is very very probably read only but these 2 operations shouldn't change the data. I don't understand why I get this message.
Sounds like the CLI does not have write permission to wherever it is trying to write command line history file .sqlite_history. Try setting the environment variable SQLITE_HISTORY (incuding file name) to a writeable location.
I have a very large sqlite database with a single table with two text columns (about 2.3 billion rows, 98GB) that I'm trying to create an index on using the sqlite3 cli tool on Ubuntu 20.04.
The command I'm trying to run is:
CREATE INDEX col1_col2_x ON tablename(col1 COLLATE NO CASE,col2);
The goal is to also create the opposite index to be able to do very fast case-insensitive searches on either column.
Every time I try, it runs for about an hour and then the process exits with just the message "Killed" and exit code 137, which I don't see listed in the sqlite3 documentation.
My first thought was running out of memory, so I tried setting the pragma temp_store_directory as well as the TEMP_DIR environment variable to same directory as the database file, which has about 8TB of free space, so I'm not sure what's going wrong.
Is sqlite not meant for databases of this size? Creating the index before insert doesn't seem to be a viable option as it's looking like it's going to take months. I should also note that I was able to create the exact same indexes successfully with a 36GB table that has the same schema so I'm wondering if I'm running into an undocumented limitation?
I'm also open to other database solutions if sqlite isn't the right solution, although preliminary tests of postgres didn't seem to be any better.
Have you considered setting any of the various PRAGMA statements for the database?
Even with a 2Gb database of only 5 million rows the following were helpful.
PRAGMA page_size = 4096;
PRAGMA cache_size = 10000;
PRAGMA synchronous = OFF;
PRAGMA auto_vacuum = FULL;
PRAGMA automatic_index = FALSE;
PRAGMA journal_mode = OFF;
page_size
Query or set the page size of the database. The page size must be a power of two between 512 and 65536 inclusive.
cache_size
Query or change the suggested maximum number of database disk pages that SQLite will hold in memory at once per open database file.
synchronous
With synchronous OFF (0), SQLite continues without syncing as soon as it has handed data off to the operating system.
automatic_vaccum
When the auto-vacuum mode is 1 or "full", the freelist pages are moved to the end of the database file and the database file is truncated to remove the freelist pages at every transaction commit.
automatic_index
Set Automatic Indexes
journal_mode
The OFF journaling mode disables the rollback journal completely. No rollback journal is ever created and hence there is never a rollback journal to delete. The OFF journaling mode disables the atomic commit and rollback capabilities of SQLite.
I have two postgresql databases, one on my local machine and one on a remote machine. If I'm connected to the local database (with psql), how do I execute a statement that inserts rows into a table on the remote database by selecting rows from a table on the local database? (I've seen this asked a handful of times, like here, here and here, but I haven't yet found a satisfactory answer or anyone saying definitively that it's not possible).
Specifically, assume I have tables remotetable and localtable, each with a single column columnA. I can run this statement successfully:
select dblink_exec('myconnection', 'insert into remotetable (columnA) values (1);');
but what I want to do is this:
select dblink_exec('myconnection', 'insert into remotetable (columnA) select columnA from localtable;');
But this fails with: relation "localtable" does not exist, presumably because localtable does not exist on the remote database.
Is it possible to do what I'm trying to do? If so, how do I indicate that localtable is, in fact, local? All of the examples I've seen for dblink-exec show inserts with static values, not with the results of a local query.
Note: I know how to query data from a remote table and insert into a local table, but I'm trying to move data in the other direction.
If so, how do I indicate that localtable is, in fact, local?
It's not possible because dblink acts as an SQL client to the remote server. That's why the queries sent through dblink_exec must be self-contained: they can do no more than any other query sent by any SQL application. Every object in the query is local to it from the server's perspective.
That is, unless you use another functionality, a Foreign-Data Wrapper with the postgres_fdw driver. This is a more sophisticated way to achieve server-to-server querying in which the SQL engine itself has this notion of foreign and local tables.
I have one process constantly insert into a sqlite3 database and another process select from the sqlite3 database(slow sql).
Does sqlite3 lock the database on reads?
I want make sure every write success. Read fail is acceptable.
According to SQLite3 locking reference after start of transaction (BEGIN command), a SHARED lock will be acquired when the first SELECT statement is executed. Shared lock means that the database may be read but not written. A RESERVED lock will be acquired when the first INSERT, UPDATE, or DELETE statement is executed.
I am looking to understand the implications of using ATTACH with databases with different read-write permissions.
I have a scenario where I need to access a large database (approx. 512MB) that resides in a read-only filesystem. There is also a small read-write database with the same schema that resides in a read-write filesystem. The read-only database provides the base data used in my scenario, with infrequent data updates stored in the read-write database.
Currently I open these two databases in separate connections and the code that maintains the connections is responsible for presenting a unified view of the data to its clients. For example, this means that the code has to merge query results from the read-only and read-write databases, etc. I realize that this setup is inelegant (and likely suboptimal) and have been looking to use the ATTACH command to create a unified view of the data in SQL rather than C++.
I am wondering then whether there are any particular gotchas related to attaching read-only and read-write databases that I should be aware of. I am looking at one of the following ATTACH scenarios:
Open the read-only database as main and ATTACH the read-write database. This is my preferred solution.
Open the read-write database as main and ATTACH the read-only database.
A third alternative?
A few google queries pointed to messages suggesting problems in scenario (1). Because I did not find a definitive answer and because my own testing using sqlite 3.6.13 did not reveal any problems, I am posting this question.
Thank you for any insights.
The documentation doesn't seem to mention any caveats with attaching read-write databases to read-only ones.
So my assumption is that what you should expect to happen were the databases to be opened separately should also happen when you open one and attach the other.
I put your scenario 1 to the test and it seems to work fine. Here is what I tried:
[someone#somewhere tmp]$ echo .dump | sqlite3 big_readonly_db
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE foo (a INT);
INSERT INTO "foo" VALUES(1);
INSERT INTO "foo" VALUES(2);
INSERT INTO "foo" VALUES(3);
INSERT INTO "foo" VALUES(4);
INSERT INTO "foo" VALUES(5);
COMMIT;
[someone#somewhere tmp]$ echo .dump | sqlite3 small_readwrite_db
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE bar (a INT);
COMMIT;
[someone#somewhere tmp]$ chmod -w big_readonly_db
[someone#somewhere tmp]$ ls -l big_readonly_db
-r--r--r-- 1 someone someone 2048 Apr 12 21:41 big_readonly_db
[someone#somewhere tmp]$ sqlite3 big_readonly_db
SQLite version 3.7.7.1 2011-06-28 17:39:05
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> attach database small_readwrite_db as rw;
sqlite> insert into bar select * from foo;
sqlite> select * from bar;
1
2
3
4
5
I know that I am quite late. But I have a similar issue and I stumbled on this thread, so here's what I currently know:
#Yakov Galka is correct and sqlite does not allow writing on attached databases if the main database is opened in readonly mode. This can be verified by using #user610650's test code but opening the database in the last shell command with sqlite3 -readonly big_readonly_db. Although I haven't found that particular behavior described in the official documentation.
But #Bill Zissimopoulos' situation is slightly different. The readonly database cannot be written due to filesystem restrictions. In this case the main database can simply be opened normally (i.e. in read/write mode) so that the attached database can be written as well. When issuing a write command on the readonly main database, sqlite still restricts it (or: isn't allowed to) with the error: attempt to write a readonly database. So the preferred solution can be used.