While a backup of a sqlite db is going on (via the .backup command), is the sqlite db locked to prevent writes?
This answer seems to suggest that the db is not locked.
sqlite3 shell command '.backup' and transaction
However, I could not find a definite source in the documentation.
According to the documentation a shared (read-only) lock is acquired.
http://www.sqlite.org/backup.html
Related
As far as I can tell both commands create a clone of the db in SQLite format in the end, so why are there two commands to do this?
.backup uses the SQLite Backup API to create a clone atomically, even if the database is in use.
.clone copies the database by just running SQL commands. As far as I can tell, no transactions are done on the source database when doing the clone, so it has a chance of getting partially updated data mid way through the clone.
A process crashed while trying to commit an insert of 2-3 million rows. Now, I'm finding the database file is locked for all activity, including reading, dumping, and running an integrity check.
Can I unlock this database in some way or get sqlite3 to recover the file?
Is there any way of recovering the contents of this database file? I do have a backup from 24 hours ago, but a variety of other data will be lost if I have to do a complete rollback to an older version of the database.
Code below:
$ sqlite3 dbFile.sqlite
SQLite version 3.19.3 2017-06-08 14:26:16
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check;
Error: database is locked
sqlite> .tables
Error: database is locked
sqlite> SELECT * FROM dbTable LIMIT 1;
Error: database is locked
Running fuser on the database file indicates that no processes are currently attempting to access the file. There is a hot file in the same directory (i.e. dbFile.sqlite-journal exists).
Using stat, I see that the folder in question is of type panfs, which seems likely to be the cause of the issue. What can I do here?
Notably not a duplicate of this question, as the database is (1) not locked by any running process, as mentioned above, (2) locked for reading as well, and (3) due to the full read/write lock the .dump solution in this answer is inapplicable (as .dump fails).
Closest related question is this question, although that one is on a different NFS filesystem.
Apparently, the PanFS server(s) did not notice the crash, and so still report the write lock.
You could try to unmount and re-mount the file system on this machine, if possible.
Alternatively, copy both files (database and journal) elsewhere. When you open that database, SQLite will roll back the partial transaction.
I downloaded and installed the SQLite ODBC Driver from http://ch-werner.de/sqliteodbc/. Works great with one issue. I can't seem to get an exclusive lock if another app has the database open.
I'm using a CAD application called Altium. I'm trying to build a prototype parts database in SQLITE to house data. I installed the ODBC driver referenced above and I can get to the data just fine. The issue is if I try to open the same database in an app like SQLITEStudio or DB Browser for SQLITE I can never get an exclusive lock to write new records. Altium never closes its connection and I don't know if that's the problem or not.
When I read https://www.sqlite.org/lockingv3.html it seems to indicate that multiple processes would need to communicate in order for one to obtain an exclusive lock (necessary for a write). This leads me to believe my issue is the first app never releasing it's shared lock so the second app can never get an exclusive lock. Am I interpreting this correctly?
I'm experiencing the same difficulties with SQLite as an Altium database. The database file can't be updated while Altium is open and had created a connection to the database. There are two workarounds I've found so far:
Enable Write Ahead Logging style database journal
The default database journal style, DELETE, creates a copy of the database, temporarily, writes to the database file, and, when the write is verified, deletes the temporary copy.
Write ahead logging, WAL, puts all changes in a temporary file while leaving the original copy of the database alone. Processes accessing the database parse the database file and the WAL temporary files, and return information reflecting all changes. I can confirm that writes outside of Altium are seen when doing a database refresh within Altium.
The writes stored in the WAL file are written to the original database when a checkpoint operation happens. Either when a process calls PRAGMA wal_checkpoint; or after a number of operations specified by the WAL Auto Checkpoint setting.
You can enable WAL journal mode in DB Browser's Edit Pragmas tab:
DB Browser Database Settings
The problem still isn't fixed, though. Checkpointing the database doesn't work while Altium is open. The database file is still locked from changes.
The caveat for us is that we want to backup and maintain a history of the database file in Git (yes, this is not a recommended use of Git). Writes to the journal files aren't tracked in Git, which doesn't see a change in the database file until it is checkpointed. This means that we can't push or pull from Git while Altium is open. We could track the database and journal files, but the other consequence is that a simple checkpoint with no data changes will be seen by Git as a change.
Use an alternative ODBC driver
I tested an alternative, proprietary ODBC driver from devart. When I configure Altium to connect to the SQLite database using the devart driver, DB Browser has no problems writing to the file while Altium is open, regardless of journal settings. Altium can also see those changes with a refresh operation.
Conclusion
Christian Werner's ODBC driver (as of version 0.9996) is locking the file in a manner that the devart driver does not, even while Atlium maintains a connection to the database through the driver. WAL journal mode can be a workaround for some situations.
I have a huge problem... I am developing desktop app with SQLite but during copy/paste process I lost a power and process was terminated so base was lost. However, I found a way to recover it but base is encrypted. When I try to open connection using conn.Open(); I get this error. If I try to open it with DB Browser for SQLite it asks me a SQLCipher encryption password so it seams to me that data is lost..
Is there any default password ?
Why did this happen and how to prevent it from happening again ?
What can I do ?
Thanks in advance.
Also check that SQLite version you're "connecting" with aligns with the DB file version.
For example, here's a DB file written by SQLite version 3+:
$ file foobar.db
foobar.db: SQLite 3.x database, last written using SQLite version 3027002
And here I also have 2 versions of sqlite:
$ sqlite -version
2.8.17
$ sqlite3 -version
3.27.2 2019-02-25 16:06:06 bd49a8271d650fa89e446b42e513b595a717b9212c91dd384aab871fc1d0alt1
Obviously in hindsight, opening foobar.db with sqlite version 2 will fail, yielding the same error message:
$ sqlite foobar.db
Unable to open database "foobar.db": file is encrypted or is not a database
But all is good with the correct version:
$ sqlite3 foobar.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite>
sqlite> .databases
main: /tmp/foobar.db
sqlite>
The error message is a catch-all, simply meaning that the file format was not recognized.
Ok, finally found a solution that works so posting the answer if anybody will have same trouble as I did..
First of all, use good recover software. For repairing the database I found 3 solutions that work without backup :
Open corrupted database using DB Browser an Export Database to SQL. Name it however you want. Then, create new database and import database from SQL.
There is software that repairs corrupted databases. Download one and use it to repair the database.
Download "sqlite3" from sqlite.org and in command line navigate to folder where "sqlite3" is unzipped. Then try to dump the entire database with .dump, and use those commands to create a new database:
sqlite3 corrupt_table_name.sqlite ".dump" | sqlite3 new.sqlite
I had the same error when I was trying to access a db dump in another system compared to compared to where it was obtained. When I tried to open on a dev machine, it threw the reported error in this thread:
$ sqlite3 db_dump.sqlite .tables
Error: file is encrypted or is not a database
This turned out to be due to the differences in the sqlite version between those systems. This dev system version was 3.6.20.
The version on the production system was 3.8.9. Once I had the sqlite3 upgraded to same version, I was able to access all its data. I am just showing below the tables are displayed as expected:
# sqlite3 -version
3.8.9
# sqlite3 db_dump.sqlite .tables
capture diskio transport
consumer filters processes
This error is rather misleading to begin with, though.
If you've interacted with the database at some point while specifying journal_mode = WAL, and then later try to use the database from a client that does not support WAL (< v3.7.0), this error can also come up.
As noted in the SQLite documentation under Backwards Compatibility, to resolve that without having to recreate the database, explicitly set the journal mode to DELETE:
PRAGMA journal_mode=DELETE;
Your database did not become encrypted (this is only one of the two options in the error message).
Your data recovery tool did not recover the correct data; what you have in the file is something else.
You have to restore the database file from the backup.
The issue is with sqlcipher version upgrade in my case, Whenever I update my pod it automatically upgrade the sqlcipher and the error occurred.
For a quick fix just manually add the SDK instead of Pod install. And for a proper solution use this link GitHub Solution
I'd like to know how the .dump command affects other applications connected to the same database. I'd like to know this for the following journal modes:
DELETE (the default mode)
WAL (write-ahead-logging)
From reading other posts on this forum .backup uses the online backup API of SQLite. It would be great to have this confirmed as well.
Thanks in advance!
The .dump command reads the contents of the database normally, just as if you would do a bunch of SELECT queries inside a transaction.
This means that when not using WAL, other connections cannot write as long as the dump is running.