SQLite transactions fail - r

Question

I'm performing thousands of calculations with R and some external software.
To keep track of this, I have constructed a pipeline in R on a SQLite3 database.
To get things done, I've set it up to allow for multiple nodes on our computing cluster to run the R script.
Naturally, we need to keep things atomized so I am actively starting transactions.
To avoid the script from crashing when trying to get a lock on the database, I have the following code that attempts to begin a transaction, and if fails, waits and then retries:
dbBeginTransaction <- function(dbconn, state='DEFERRED', timeout=5, retries=10) {
state <- toupper(state)
if (!state %in% c('DEFERRED','IMMEDIATE','EXCLUSIVE')) stop('Attempt at illegal transaction.')
res <- NULL
exit <- FALSE
for (i in 1:retries) {
try(
res <- dbSendQuery(dbconn, paste('BEGIN',state,'TRANSACTION;'))
, silent=TRUE
)
# res is null if above query fails.
err <- dbGetException(dbconn)
if (err$errorNum == 0) { ## OK
#return(TRUE)
exit <- TRUE
break
} else if (err$errorNum == 5) { ## Database locked.
if (i == retries+1) {
cat('Database still locked after',i,'attempts.\n',file=stderr())
#return(FALSE)
exit <- FALSE
break
} else {
Sys.sleep(timeout)
}
} else {
## errorNum == 1 ## Already within transaction.
cat(err$errorMsg, '\n', file = stderr())
#return(FALSE)
exit <- FALSE
break
}
}
invisible(exit)
}
## Usage:
insert_results <- function(results) {
## Some preparing of results
if (dbBeginTransaction(conn, 'EXCLUSIVE') == FALSE) return(FALSE)
dbSendPreparedQuery(conn, 'INSERT INTO results (...) VALUES (...);', results)
dbCommit(conn)
}
## After a computation:
results <- magic()
if (!insert_results(results)) stop('Could not save results')
When debugging, it works as it should. But every now and then I get this odd error, and the script crashes:
Error in sqliteSendQuery(conn, statement, bind.data) :
rsqlite_query_send: could not execute: database is locked
Calls: dbSendPreparedQuery ... dbSendPreparedQuery -> .local -> sqliteSendQuery -> .Call
Execution halted
I cannot fathom what happens, and I am yet to reproduce the error.
The error is clear, but I would have thought that my routine had prevented it.
Any thoughts on why this is happening?
I'm running R under linux, as seen by:
> sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US LC_NUMERIC=C LC_TIME=en_US
[4] LC_COLLATE=en_US LC_MONETARY=en_US LC_MESSAGES=en_US
[7] LC_PAPER=en_US LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_US LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] RSQLite_1.0.0 DBI_0.3.1

Answer 1

Encountered this problem today in the office today. This is the suggestion of my colleague: It can be solved by making sure that every time you do a database connection and make some sort of change to the database you do a dbCommit(con). The new code would look like this:
dbBeginTransaction <- function(dbconn, state='DEFERRED', timeout=5, retries=10) {
state <- toupper(state)
if (!state %in% c('DEFERRED','IMMEDIATE','EXCLUSIVE')) stop('Attempt at illegal transaction.')
res <- NULL
exit <- FALSE
for (i in 1:retries) {
try(
dbBegin(dbconn)
res <- dbSendQuery(dbconn, paste('BEGIN',state,'TRANSACTION;'))
, silent=TRUE
)
# res is null if above query fails.
err <- dbGetException(dbconn)
if (err$errorNum == 0) { ## OK
#return(TRUE)
exit <- TRUE
break
} else if (err$errorNum == 5) { ## Database locked.
if (i == retries+1) {
cat('Database still locked after',i,'attempts.\n',file=stderr())
#return(FALSE)
exit <- FALSE
break
} else {
Sys.sleep(timeout)
}
} else {
## errorNum == 1 ## Already within transaction.
cat(err$errorMsg, '\n', file = stderr())
#return(FALSE)
exit <- FALSE
break
}
}
invisible(exit)
}
## Usage:
insert_results <- function(results) {
## Some preparing of results
if (dbBeginTransaction(conn, 'EXCLUSIVE') == FALSE) return(FALSE)
dbSendPreparedQuery(conn, 'INSERT INTO results (...) VALUES (...);', results)
dbCommit(conn)
}
## After a computation:
results <- magic()
if (!insert_results(results)) stop('Could not save results')
And I solved by adding dbBegin(con) just before your begin any transactions with the database
dbBeginTransaction <- function(dbconn, state='DEFERRED', timeout=5, retries=10) {
state <- toupper(state)
if (!state %in% c('DEFERRED','IMMEDIATE','EXCLUSIVE')) stop('Attempt at illegal transaction.')
res <- NULL
exit <- FALSE
for (i in 1:retries) {
try(
res <- dbSendQuery(dbconn, paste('BEGIN',state,'TRANSACTION;'))
, silent=TRUE
dbCommit(dbconn)
)
# res is null if above query fails.
err <- dbGetException(dbconn)
if (err$errorNum == 0) { ## OK
#return(TRUE)
exit <- TRUE
break
} else if (err$errorNum == 5) { ## Database locked.
if (i == retries+1) {
cat('Database still locked after',i,'attempts.\n',file=stderr())
#return(FALSE)
exit <- FALSE
break
} else {
Sys.sleep(timeout)
}
} else {
## errorNum == 1 ## Already within transaction.
cat(err$errorMsg, '\n', file = stderr())
#return(FALSE)
exit <- FALSE
break
}
}
invisible(exit)
}
## Usage:
insert_results <- function(results) {
## Some preparing of results
if (dbBeginTransaction(conn, 'EXCLUSIVE') == FALSE) return(FALSE)
dbSendPreparedQuery(conn, 'INSERT INTO results (...) VALUES (...);', results)
dbCommit(conn)
}
## After a computation:
results <- magic()
if (!insert_results(results)) stop('Could not save results')