I'm finding that new sqlite3 database files are locked before any use that I am aware of.
sqlite3 new.sqlite
sqlite> SELECT * FROM SQLITE_MASTER;
Error: database is locked
lsof on the new file is empty. Copying the database file to a new location doesn't help. Permissions on the file are OK.
How else can I determine why a new sqlite3 file might be locked?
Looking at the docs, my best guess is that this is occuring because the database file is on an NFS mount for Vagrant. According to the docs:
One should note that POSIX advisory locking is known to be buggy or
even unimplemented on many NFS implementations ... Your best defense
is to not use SQLite for files on a network filesystem.
https://www.sqlite.org/lockingv3.html
I was able to resolve the issue by setting file permissions on the mounted folder on the host machine.
Related
A process crashed while trying to commit an insert of 2-3 million rows. Now, I'm finding the database file is locked for all activity, including reading, dumping, and running an integrity check.
Can I unlock this database in some way or get sqlite3 to recover the file?
Is there any way of recovering the contents of this database file? I do have a backup from 24 hours ago, but a variety of other data will be lost if I have to do a complete rollback to an older version of the database.
Code below:
$ sqlite3 dbFile.sqlite
SQLite version 3.19.3 2017-06-08 14:26:16
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check;
Error: database is locked
sqlite> .tables
Error: database is locked
sqlite> SELECT * FROM dbTable LIMIT 1;
Error: database is locked
Running fuser on the database file indicates that no processes are currently attempting to access the file. There is a hot file in the same directory (i.e. dbFile.sqlite-journal exists).
Using stat, I see that the folder in question is of type panfs, which seems likely to be the cause of the issue. What can I do here?
Notably not a duplicate of this question, as the database is (1) not locked by any running process, as mentioned above, (2) locked for reading as well, and (3) due to the full read/write lock the .dump solution in this answer is inapplicable (as .dump fails).
Closest related question is this question, although that one is on a different NFS filesystem.
Apparently, the PanFS server(s) did not notice the crash, and so still report the write lock.
You could try to unmount and re-mount the file system on this machine, if possible.
Alternatively, copy both files (database and journal) elsewhere. When you open that database, SQLite will roll back the partial transaction.
Where is the SQLite database stored i.e. directory path on windows 7 when created ?
A SQLite database is a regular file. It is created in your script current directory.
.databases
If you run this command inside SQLite
.databases
it lists the path of all currently connected databases. Sample output:
seq name file
--- --------------- ----------------------------------------------------------
0 main /home/me/a.db
There is no "standard place" for a sqlite database. The file's location is specified to the library, and may be in your home directory, in the invoking program's folder, or any other place.
If it helps, sqlite databases are, by convention, named with a .db file extension.
If you are running Rails (its the default db in Rails) check the {RAILS_ROOT}/config/database.yml file and you will see something like:
database: db/development.sqlite3
This means that it will be in the {RAILS_ROOT}/db directory.
When you call sqlite3_open() you specify the filepath the database is opened from/saved to, if it is not an absolute path it is specified relative to your current working directory.
It depends on how you initialized the database. If you used the command line shell for SQLite (e.g. sqlite3 ex1) to create the database, it'll be a path from the root of your local machine. If you used a Python script to create the database, it'll be a path from your project.
To check the former, run the command line shell:
sqlite3
sqlite> .databases
To check the path in your project, you can print the path in the connection. For example:
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATABASE = 'db'
def get_db_connection():
print(os.path.join(BASE_DIR, DATABASE, "database.db"))
conn = sqlite3.connect(os.path.join(BASE_DIR, DATABASE, "database.db"))
conn.row_factory = sqlite3.Row
return conn
In my case I think it was an access issue. I saved the SQLite files to "C:/Program Files (x86)/sqlite". I CD'd there, ran sqlite3, and created a database called test.db:
As you can see, I ran .database, which told me the .db file was created in the same directory, so I went to confirm in File Explorer, and it wasn't there:
Curiously the database was working correctly in spite of this.
It was only through trial-and-error that I discovered that I could save in some locations, but not others. It appears to me that SQLite can't save to locations that require elevation. In my case, moving from Program Files to My Documents made the issue go away.
I find it quite irritating that SQLite doesn't just tell me "access denied" instead of trying to be clever and saving to some location that I can't even find.
In Windows machines (Windows 2010), by default, the new SQLite database files will be stored in the same folder where Sqlite3.EXE application is stored in your machine. However , we can create a new folder in Windows and within sqlite> prompt, you may use the .cd to change to the new working directory.
It is a good idea to give a .db file extension to the new database files that you create (even though it is not mandatory to have any file extension)
The SQLite command, .databases will show the default database "main" or currently created or currently opened database or all "attached" database files with file path. The .attach is useful to attach more than one database file to the current connection when we want to work with tables belonging to different databases.
Regards,
Biju Joseph N.,
Houston TX, USA (January 12, 2023)
the database path will be displayed, when using .databases
SQLite is created in your python directory where you installed the python.
SQLit Database is simply a file where your local data is stored on your local machine
In Windows 10 if in the prompt command the path where you start sqlite is
C:\users\USER_NAME
You can find it in the user home folder.
The .db file is stored where you start the sqlite command.
I hope this solve the issue
I have a small Perl-based CGI application, which I am running in the project web space provided for a SourceForge project. This application stores data in a SQLite (v. 3) database file.
When I run test scripts from a shell, I can read from and write to this SQLite file. However, when the CGI code is executed by Apache, it has read-only access. Write operations result in a log file error:
error.log.web-2:[Wed Oct 27 14:40:22 2010] [error] [client 127.0.0.1] DBD::SQLite::db do failed: unable to open database file
For testing purposes, I have cranked the permissions for that SQLite file all the way up to 777. No difference.
However, there are some funny caveats to SourceForge's project web space, and I wonder if I'm being tripped up by that. Generally, the main web server filesystem is read-only to Apache. If you have files that need to be writable at runtime, you're supposed to store them in a special "persistent" directory elsewhere... and create symlinks from your web space to the actual files under that directory.
I have done this, and the permissions are set to 777 for both the symlink and the actual SQLite file under the "persistence" location. I know this mechanism works in general, because I'm doing the same thing with cache and log files and it works there.
I'm wondering if there's anything funky about SQLite itself, along the lines of it not wanting to open a symlink (rather than a raw file) for writing.
I believe the answer to this question is that it can't be done. Further research into SQLite tells me that the driver must get a lock on the database file before it can do any write operations. This type of lock cannot be obtained when the actual file is on a different machine with its filesystem cross-mounted.
I believe this is the case with SourceForge project web space hosting. It looks like the (writable) "persistent" directory is actually on a totally separate machine from the read-only web server filesystem.
In short, if you stumble across this question because you're having the same issue... either look for different web space hosting, or else it may be time to re-work your app and step up to MySQL or some other DB (SourceForge gives you free MySQL hosting anyway).
Another issue is if you have permissions for the specific db file but you don't have permission to make the temporary files in the directory. (Mixed permissions, or too restrictive permissions)
https://www.sqlite.org/tempfiles.html
If you can't write the temporary files then you can't do any writes on a sqlite database file. If you switch it to a :memory: database you could get by or maybe use the pragma mentioned by #bob.faist PRAGMA temp_store = MEMORY, but really you should diagnosis and fix the permissions problem if possible.
Use these commands to see if you have permission to write in those file locations.
ls -l app.db
getfacl app.db
ls -l -d . # check the directory to see if you can write the temp files there
getfacl .
Use chmod or setfacl -m to fix the files or folders to let you write to them.
Also check your diskspace.
df -k
If it shows that your partition where the database file is located or is trying to write its files to are full, you could also get these kinds of issues.
Hope that helps.
I have an uninstall issue with an app which is using sqlite: during installation a blank sqlite db is created in [CommonAppData]\MyApp\mydb.sqlite, e.g. C:\Documents and Settings\All Users\Application Data\MyApp\mydb.sqlite. When I uninstall my app it can't delete the sqlite db, despite it removing the applications that are connectng to it. Using process explorer I can see that it's explorer.exe that has a lock on the MyApp folder (not on the sqlite file).
I've not seen this sort of thing before. Is it possible this is caused by the app not correctly closing/disposing of connections? I understand that at some level windows manages the fact that several threads & processes are accessing my db file, and it handles the locking. Is it possible that if my app isn't closing connections etc correctly then windows gets confused about whether the file is locked or not?
Or is that not possible and it must simply be something wrong with my MSI?
thanks for any suggestions!
UPDATE: not only can I not delete the folder or file, if I create a new file in that folder (e.g. a new txt doc) I can then not delete that file! So it's some wacky lock on the folder....
UPDATE: actually...it might just be permissions on that folder! In my msi i was setting permissions on that folder, and I think I didn't give delete rights so when I uninstalled I didn't have access to delete it :-/
Use handle.exe from the SysInternals collection to find out what has the remaining handle to the file.
It could also be your MSI, so check you are doing things in the right order by doing;
msiexec /u mymsi.msi /lv* mylog.txt
How can I change an SQLite database from read-only to read-write?
When I executed the update statement, I always got:
SQL error: attempt to write a readonly database
The SQLite file is a writeable file on the filesystem.
There can be several reasons for this error message:
Several processes have the database open at the same time (see the FAQ).
There is a plugin to compress and encrypt the database. It doesn't allow to modify the DB.
Lastly, another FAQ says: "Make sure that the directory containing the database file is also writable to the user executing the CGI script." I think this is because the engine needs to create more files in the directory.
The whole filesystem might be read only, for example after a crash.
On Unix systems, another process can replace the whole file.
I solved this by changing owner from "root" to my own user, on all files in Database's folder.
Just do ls -l on said folder, and if any of the files is owned by root, just change it to your user, like:
# For each file do:
sudo chown "$USER":"$USER" /path/to/my-folder/file.txt
# Or "R"ecursive.
sudo chown -R "$USER":"$USER" /path/to/my-folder
(this error message is typically misleading, and is usually a general permissions error)
On Windows
If you're issuing SQL directly against the database, make sure whatever application you're using to run the SQL is running as administrator
If an application is attempting the update, the account that it uses to access the database may need permissions on the folder containing your database file. For example, if IIS is accessing the database, the IUSR and IIS_IUSRS may both need appropriate permissions (you can try this by temporarily giving these accounts full control over the folder, checking if this works, then tying down the permissions as appropriate)
This error usually happens when your database is accessed by one application already, and you're trying to access it with another application.
To share personal experience I encountered with this error that eventually fix both. Might not necessarily be related to your issue but it appears this error is so generic that it can be attributed to gazillion things.
Database instance open in another application. My DB appeared to have been in a "locked" state so it transition to read only mode. I was able to track it down by stopping the a 2nd instance of the application sharing the DB.
Directory tree permission - please be sure to ensure user account has permission not just at the file level but at the entire upper directory level all the way to / level.
Thanks
If using Android.
Make sure you have added the permission to write to your EXTERNAL_STORAGE to your AndroidManifest.xml.
Add this line to your AndroidManifest.xml file above and outside your <application> tag.
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
This will allow your application to write to the sdcard. This will help if your EXTERNAL_STORAGE is where you have stored your database on the device.
On win10 after a system crash, try to open db with DB Browser, but read only.
Simply delete the journal file.
In Linux command shell, I did:
chmod 777 <db_folder>
Where contains the database file.
It works. Now I can access my database and make insert queries.
On Windows:
tl;dr: Try opening the file again.
Our system was suffering this problem, and it definitely wasn't a permissions issue, since the program itself would be able to open the database as writable from many threads most of the time, but occasionally (only on Windows, not on OSX), a thread would get these errors even though all the other threads in the program were having no difficulties.
We eventually discovered that the threads that were failing were only those that were trying to open the database immediately after another thread had closed it (within 3 ms). We speculated that the problem was due to the fact that Windows (or the sqlite implementation under windows) doesn't always immediately clean up up file resources upon closing of a file. We got around this by running a test write query against the db upon opening (e.g., creating then dropping a table with a silly name). If the create/drop failed, we waited for 50 ms and tried again, repeating until we succeeded or 5 seconds elapsed.
It worked; apparently there just needed to be enough time for the resources to flush out to disk.
On Ubuntu, change the owner to the Apache group and grant the right permissions (no, it's not 777):
sudo chgrp www-data <path to db.sqlite3>
sudo chmod 664 <path to db.sqlite3>
Update
You can set the permissions for group and user as well.
sudo chown www-data:www-data <path to db.sqlite3>
If <db_name>.sqlite-journal file exists in the same folder with DB file, that means your DB is opened currently and in the middle of some changes (or it had been at the moment when DB folder was copied). If you try to open DB at this moment error attempt to write a readonly database (or similar) could appear.
As a solution, wait till <db_name>.sqlite-journal disappears or remove it (is not recommended on the working system)
I had this problem today, too.
It was caused by ActiveSync on Windows Mobile - the folder I was working in was synced so the AS process grabbed the DB file from time to time causing this error.
On Linux, give read/write permissions to the entire folder containing the database file.
Also, SELinux might be blocking the write. You need to set the correct permissions.
In my SELinux Management GUI (on Fedora 19), I checked the box on the line labelled httpd_unified (Unify HTTPD handling of all content files), and I was good to go.
I'm using SQLite on ESP32 and all answers here are "very strange"....
When I look at the data on the flash of the ESP I notice there is only one file for the whole db (there is also a temp file).
In this db file we have of course the user tables but also the system tables so "sqlite_master" for example which contain the definiton of the tables.
So, it's seems hard to belive this can be a "chmod" problem, because if the file is read only, even creating table would be impossible as SQLite would be unable to write the "sqlite_master" data...
So I think our friend user143482 is trying to acesse a "read only" table. In SQLite source code we can see a function named tabIsReadOnly with this comment:
/* Return true if table pTab is read-only.
**
** A table is read-only if any of the following are true:
**
** 1) It is a virtual table and no implementation of the xUpdate method
** has been provided
**
** 2) It is a system table (i.e. sqlite_master), this call is not
** part of a nested parse and writable_schema pragma has not
** been specified
**
** 3) The table is a shadow table, the database connection is in
** defensive mode, and the current sqlite3_prepare()
** is for a top-level SQL statement.
*/