sqlite3_prepare function fails if DB is opened with SQLITE_OPEN_READONLY.
Error message: error #10: disk I/O error
Sqlite extended error code: SQLITE_IOERR_LOCK (3850)
Errno: EBADF 9 /* Bad file number */
Everything works fine if DB is opened with SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE |
SQLITE_OPEN_FULLMUTEX;
Any ideas on what may cause this problem?
rc=sqlite3_open_v2("example.db",&db,SQLITE_OPEN_READONLY,0);
sqlite3_busy_timeout(db,1000);
selectQuery = "select * from test;";
rc = sqlite3_prepare(db, selectQuery, strlen(selectQuery), &stmt, 0);
if(rc!=SQLITE_OK)
{
printf("sql error #%d: %s", rc, sqlite3_errmsg(db));
printf( "SQL ext error: %d\n", sqlite3_extended_errcode(db));
printf( "errno: %d\n", errno );
}
The problem is in OS lock implementation. Fcntl() doesn't allow to lock file if it is opened with read only permissions. As a result - SQLite can't do anything in read only mode.
Related
On Airflow 2.2.4 (Postgres and Celery).
I have a connection created for Microsoft SQL Server. When I try to test the connection, UI asked me to enter API Auth user/password (Basic Auth) and showed up a green flash "Connection successfully tested"
But,
When I use the same connection ID in an Operator definition to run some SQL queries, I am getting an error as below.
pymssql._mssql.MSSQLDatabaseException: (20009, b'DB-Lib error message 20009, severity 9:\nUnable to connect: Adaptive Server is unavailable or does not exist (<IP>)\nNet-Lib error during Connection timed out (110)\n
DB-Lib error message 20009, severity 9:\nUnable to connect: Adaptive Server is unavailable or does not exist (<IP>)\n
Net-Lib error during Connection timed out (110)\n')
Here is what my code from my custom operator looks like,
def get_update_num(self):
conn = None
try:
query = f"""
UPDATE SOME_TABLE
SET COL_A = COL_A+1
OUTPUT INSERTED.COL_A
WHERE COL_ID='12345'
self.log.info(f"SQL = {query}")
conn_id = self.conn_id # This is equal to MSSQL_CONNECTION
hook = MsSqlHook(mssql_conn_id=conn_id)
conn = hook.get_conn()
hook.set_autocommit(conn, True)
cursor = conn.cursor()
cursor.execute(query)
row = cursor.fetchone()
self.log.info(f"row = {row}")
return row[0]
except Exception as e:
message = "Error: Could not run SQL"
raise AirflowException(message)
finally:
if not conn:
conn.close()
Any help would be much appreceated.
I am using Apache Spark 1.5.1 and trying to connect to a local SQLite database named clinton.db. Creating a data frame from a table of the database works fine but when I do some operations on the created object, I get the error below which says "SQL error or missing database (Connection is closed)". Funny thing is that I get the result of the operation nevertheless. Any idea what I can do to solve the problem, i.e., avoid the error?
Start command for spark-shell:
../spark/bin/spark-shell --master local[8] --jars ../libraries/sqlite-jdbc-3.8.11.1.jar --classpath ../libraries/sqlite-jdbc-3.8.11.1.jar
Reading from the database:
val emails = sqlContext.read.format("jdbc").options(Map("url" -> "jdbc:sqlite:../data/clinton.sqlite", "dbtable" -> "Emails")).load()
Simple count (fails):
emails.count
Error:
15/09/30 09:06:39 WARN JDBCRDD: Exception closing statement
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection is closed)
at org.sqlite.core.DB.newSQLException(DB.java:890)
at org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109)
at org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1$$anonfun$8.apply(JDBCRDD.scala:358)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1$$anonfun$8.apply(JDBCRDD.scala:358)
at org.apache.spark.TaskContextImpl$$anon$1.onTaskCompletion(TaskContextImpl.scala:60)
at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:79)
at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:77)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:77)
at org.apache.spark.scheduler.Task.run(Task.scala:90)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
res1: Long = 7945
I got the same error today, and the important line is just before the exception:
15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection is closed)
at org.sqlite.core.DB.newSQLException(DB.java:890)
at org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109)
at org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
So Spark succeeded to close the JDBC connection, and then it fails to close the JDBC statement
Looking at the source, close() is called twice:
Line 358 (org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD, Spark 1.5.1)
context.addTaskCompletionListener{ context => close() }
Line 469
override def hasNext: Boolean = {
if (!finished) {
if (!gotNext) {
nextValue = getNext()
if (finished) {
close()
}
gotNext = true
}
}
!finished
}
If you look at the close() method (line 443)
def close() {
if (closed) return
you can see that it checks the variable closed, but that value is never set to true.
If I see it correctly, this bug is still in the master. I have filed a bug report.
Source: JDBCRDD.scala (lines numbers differ slightly)
I wrote simple pro*c program to check database connectivity. The code is :
int main()
{
char *conn_string = "IDA/IDA#DBISPSS";
int x = 10;
printf("value of x is before db connection %d\n",x);
printf(" conn_string %s \n",conn_string);
EXEC SQL CONNECT :conn_string;
EXEC SQL SELECT 1 INTO :x FROM DUAL;
printf("value of x is %d\n",x);
return 0;
}
Following commands I executed to create exectuable (test_connection) of pro*c code
proc test_connection.pc
cc -I${ORACLE_HOME}/precomp/public -c test_connection.c
cc test_connection.o -o test_connection -L$ORACLE_HOME/lib -lclntsh
and when I executed test_connection exe,the output is
value of x is before db connection 10
conn_string IDA/IDA#DBISPSS
Segmentation fault
But the same code workes well in another linux machine and solaris machine.
Why segmentation fault is thrown?
I tested in HPUX 11.11/Oracle 11 and work ok. I don't see any problem, but try some changes:
Declare 'x' into a DECLARE SECTION:
EXEC SQL BEGIN DECLARE SECTION;
int x = 0;
EXEC SQL END DECLARE SECTION;
Try this connection command:
EXEC SQL BEGIN DECLARE SECTION;
char *user = "abc", *password = "123", *database="base";
EXEC SQL END DECLARE SECTION;
EXEC SQL DECLARE BASE_HANDLE DATABASE;
...
EXEC SQL CONNECT :user IDENTIFIED BY :password AT BASE_HANDLE USING :database;
...
EXEC SQL AT BASE_HANDLE SELECT 1...
Insert a printf("here 1"); between EXEC SQL CONNECT... and EXEC SQL SELECT ... to see where SEGFAULT is thrown.
I had that problem and no amount of fiddling with my source made any difference. What finally worked was when I reinitialized all (ALL) my libraries to make sure that Oracle only had access to the 32 bit versions of the library. It seems Oracle was somehow getting connected to a 64 bit library. Only by removing all references to any libraries or executables except the 32 bit versions worked. This included running a 32 bit version of Pro*C.
I have worked little bit in OpenCL now but recently "clBuildProgram" failed in one of my program. My code excerpt is below:
cl_program program;
program = clCreateProgramWithSource(context, 1, (const char**) &kernel_string, NULL, &err);
if(err != CL_SUCCESS)
{
cout<<"Unable to create Program Object. Error code = "<<err<<endl;
exit(1);
}
if(clBuildProgram(program, 0, NULL, NULL, NULL, NULL) != CL_SUCCESS)
{
cout<<"Program Build failed\n";
size_t length;
char buffer[2048];
clGetProgramBuildInfo(program, device_id[0], CL_PROGRAM_BUILD_LOG, sizeof(buffer), buffer, &length);
cout<<"--- Build log ---\n "<<buffer<<endl;
exit(1);
}
Normally earlier I got syntax or other errors inside kernel file here with the help of "clGetProgramBuildInfo()" function whenever "clBuildProgram" Failed but when this program runs, on console it only prints:
Program Build failed
--- Build log ---
And when I tried to print the error code returned by "clBuildProgram"; it is "-11"......
What can be the problem with my kernel file that I dont get any build fail information ?
You can learn the meaning of OpenCL error codes by searching in cl.h. In this case, -11 is just what you'd expect, CL_BUILD_PROGRAM_FAILURE. It's certainly curious that the build log is empty. Two questions:
1.) What is the return value from clGetProgramBuildInfo?
2.) What platform are you on? If you are using Apple's OpenCL implementation, you could try setting CL_LOG_ERRORS=stdout in your environment. For example, from Terminal:
$ CL_LOG_ERRORS=stdout ./myprog
It's also pretty easy to set this in Xcode (Edit Scheme -> Arguments -> Environment Variables).
If you are using the C instead of C++:
err = clBuildProgram(program, 0, NULL, NULL, NULL, NULL);
////////////////Add the following lines to see the log file///////////
if (err != CL_SUCCESS) {
char *buff_erro;
cl_int errcode;
size_t build_log_len;
errcode = clGetProgramBuildInfo(program, devices[0], CL_PROGRAM_BUILD_LOG, 0, NULL, &build_log_len);
if (errcode) {
printf("clGetProgramBuildInfo failed at line %d\n", __LINE__);
exit(-1);
}
buff_erro = malloc(build_log_len);
if (!buff_erro) {
printf("malloc failed at line %d\n", __LINE__);
exit(-2);
}
errcode = clGetProgramBuildInfo(program, devices[0], CL_PROGRAM_BUILD_LOG, build_log_len, buff_erro, NULL);
if (errcode) {
printf("clGetProgramBuildInfo failed at line %d\n", __LINE__);
exit(-3);
}
fprintf(stderr,"Build log: \n%s\n", buff_erro); //Be careful with the fprint
free(buff_erro);
fprintf(stderr,"clBuildProgram failed\n");
exit(EXIT_FAILURE);
}
I encountered the same problem with an empty log file. I was testing my ocl kernel on a different computer. It had 2 platforms instead of one. One Intel GPU and one AMD GPU. I only had AMD OCL SDK installed. Installing the Intel OCL SDK fixed the problem. Also selecting the AMD platform instead of the Intel GPU platform fixed it.
I've seen this happen on OSX 10.14.6 when the OpenCL kernel source is missing the _kernel attribute tag. If both the _kernel tag and return type are missing it seems to crash the system OpenCL compiler daemon, which then takes a few seconds to restart before new kernels will compile again.
I am working on embedded system and the device has linux kernel with sqlite database. Wanted to know if the sqlite database can be partitioned with secure and normal partitions.
How can the encryption be achieved for sqlite database file in linux.
Maybe I am too late to answer this question, but I was facing this issue from couple of days and couldn't find any solid solution online. I have found solution hence I am sharing it.
//Steps to make sqlite database authenticated
download sqlite3 amalgamation zip file
unzip the file. The file should contain shell.c, sqlite3.c, sqlite3.h, sqlite3ext.h
click on find the link here
3a. Open userauth.c and copy the entire code and paste it at the end of your sqlite3.c file.
3b. Open sqlite3userauth.h and copy the entire code and pase it at the end of your sqlite3.h file.
create a output file for executing the command in shell
command: gcc -o sqlite3Exe shell.c sqlite3.c -DSQLITE_USER_AUTHENTICATION -ldl -lpthread
4a. Youll get error no such file "sqlite3userauth.h" in your shell.c file:
solution: go to that file and comment th line.(this is because youve already included the necessary code when you copied sqlite3auth.h into sqlite3.h)
4b. Test your output file by running ./sqlite3Exe (this is the name youve given to the output file generated in previous step). you'll get sqlite console.
4c. Create a database and on the authentication flag:
command1: .open dbname.db
command2: .auth on
command3: .exit//command 3 is optional
Building the library
5a: creating object file
//Compiling sqlite3.c to create object after appending our new code
command: gcc -o sqlite3.o -c sqlite3.c -DSQLITE_USER_AUTHENTICATION
With this command, we generate object file which we can use to compile our c file.
Create c file to authenticate your database:
//authUser.c
#include "stdio.h"
#include "stdlib.h"
#include "sqlite3.h"
int main(int argc,char * argv[]){
int a = 10;
int rtn, rtn2;
sqlite3 *db;
char *sql, *zErMsg;
rtn = sqlite3_open("dbname.db", &db);
rtn = sqlite3_user_add(db,"username","password",2, 1);//last but one param is for number of bytes for password, last param is for weather the user is admin or not
if(rtn){
fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));
return(0);
}else{
fprintf(stderr, "Protected database successfully\n");
}
sqlite3_close(db);
return 0;
}
Compiling the program
//Compiling the program
command1: gcc authUser.c sqlite3.o -lpthread -ldl
command2: ./a.out //Output:protected database successfully
create c file to create table if the user is authenticated
//createTable.c
#include "stdio.h"
#include "stdlib.h"
#include "sqlite3.h"
static int callback(void *NotUsed, int argc, char **argv, char **azColName){
int i;
for(i=0; i less then argc; i++){
printf("%s = %s\n", azColName[i], argv[i] ? argv[i] : "NULL");
}
printf("\n");
return 0;
}
int main(int argc,char * argv[]){
int a = 10;
int rtn, rtn2;
sqlite3 *db;
char *sql, *zErMsg;
rtn = sqlite3_open("dbname.db", &db);
rtn = sqlite3_user_authenticate(db, "user","password",2);
if(rtn){
fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));
return(0);
}else{
fprintf(stderr, "Opened database successfully\n");
}
sql = "create table newtable(id int not null primary key, name varchar(100) not null)";
//sql = "insert into newtable values(5, 'ishwar')";
rtn = sqlite3_exec(db, sql, callback, 0, &zErMsg);
if(rtn != SQLITE_OK){
sqlite3_free(zErMsg);
}else{
fprintf(stdout, "Table created successfully \n");
//fprintf(stdout, "inserted successfully \n");
}
sqlite3_close(db);
return 0;
}`
compiling the program
//Compiling the program
command1: gcc createTable.c sqlite3.o -lpthread -ldl
command2: ./a.out //Output:Table created successfully
Create c file to add values in table
from the previous code, you can see two sql variable and two fprintf inside else, now uncomment the commented line and comment the other one. and runt the same command as above
output: Inserted successfully
And youre done, try experimenting with the code, change the values of sqlite3_user_authenticate function you wont be able to do these operations,at max you may be able to open database(when you comment the sqlite3_user_authenticate functon.nothing else)
Testing it with shell
Run the command: ./sqlite3Exe (the output file we created in step 4)
command1: .open dbname.db
command2: .tables //you should get error, user_auth
Thank you(please feel free to mail me in case of any problem: ishwar.rimal#gmail.com)
for encryption to be achieved with SQLite, you need to license some extensions from the SQLite author.
http://www.sqlite.org/support.html
David Segleau here, Director of Product Management for Berkeley DB.
The recent 5.1 release of Oracle Berkeley DB (5.1.7) integrates the Berkeley DB encryption feature with the SQLite-based SQL API. You can read about it here.