How to compile sqlite3 extension - CSV virtual table - sqlite

Question

I am trying to use the CSV Virtual Table extension for sqlite3. I get stuck on the first step of compiling the extension on a Mac (MacOS High Sierra 10.13.6).
I downloaded the source code for csv.c from this page. I also grabbed the sqlite source code amalgamation from here.
I used the following command to compile:
gcc -g -fPIC -dynamiclib csv.c -o csv.dylib
However, I get the following error:
csv.c:115:3: error: no member named '__builtin___vsnprintf_chk' in 'struct sqlite3_api_routines'
sqlite3_vsnprintf(CSV_MXERR, p->zErr, zFormat, ap);
^~~~~~~~~~~~~~~~~
/usr/include/sqlite3ext.h:437:53: note: expanded from macro 'sqlite3_vsnprintf'
#define sqlite3_vsnprintf sqlite3_api->vsnprintf
~~~~~~~~~~~ ^
/usr/include/secure/_stdio.h:75:3: note: expanded from macro 'vsnprintf'
__builtin___vsnprintf_chk (str, len, 0, __darwin_obsz(str), format, ap)
^
csv.c:115:21: warning: incompatible integer to pointer conversion passing 'int' to parameter of type 'const void *' [-Wint-conversion]
sqlite3_vsnprintf(CSV_MXERR, p->zErr, zFormat, ap);
^~~~~~~~~
csv.c:67:19: note: expanded from macro 'CSV_MXERR'
#define CSV_MXERR 200
^~~
/usr/include/secure/_stdio.h:75:57: note: expanded from macro 'vsnprintf'
__builtin___vsnprintf_chk (str, len, 0, __darwin_obsz(str), format, ap)
^~~
/usr/include/secure/_common.h:39:54: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^~~~~~
csv.c:568:5: error: use of undeclared identifier 'sqlite3_str'
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:568:18: error: use of undeclared identifier 'pStr'
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:568:25: warning: implicit declaration of function 'sqlite3_str_new' is invalid in C99 [-Wimplicit-function-declaration]
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:571:5: warning: implicit declaration of function 'sqlite3_str_appendf' is invalid in C99 [-Wimplicit-function-declaration]
sqlite3_str_appendf(pStr, "CREATE TABLE x(");
^
csv.c:571:25: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, "CREATE TABLE x(");
^
csv.c:581:29: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, "%sc%d TEXT", zSep, iCol);
^
csv.c:588:31: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr,"%s\"%w\" TEXT", zSep, z);
^
csv.c:597:31: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr,"%sc%d TEXT", zSep, ++iCol);
^
csv.c:603:25: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, ")");
^
csv.c:604:18: warning: implicit declaration of function 'sqlite3_str_finish' is invalid in C99 [-Wimplicit-function-declaration]
CSV_SCHEMA = sqlite3_str_finish(pStr);
^
csv.c:604:37: error: use of undeclared identifier 'pStr'
CSV_SCHEMA = sqlite3_str_finish(pStr);
^
csv.c:643:27: error: use of undeclared identifier 'SQLITE_VTAB_DIRECTONLY'
sqlite3_vtab_config(db, SQLITE_VTAB_DIRECTONLY);
^
4 warnings and 10 errors generated.
What am I doing wrong?

Answer 1

Richard Hipp at sqlite.org posted a solution on how to compile an SQLite extension.
The script for compiling the CSV extension looks like this (based on https://www.sqlite.org/loadext.html and https://github.com/sqlite/sqlite/blob/master/README.md):
wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
mkdir bld
cd bld
../sqlite/configure
make
gcc -g -I. -fPIC -dynamiclib ../sqlite/ext/misc/csv.c -o csv.dylib
Testing the script:
echo -e 'col_text,col_int\napples,3\noranges,5' > sample.csv
./sqlite3 '' '.load csv' 'CREATE VIRTUAL TABLE temp.t1 USING csv(filename="sample.csv");' 'SELECT * FROM t1;'
That's it.
You can also try compiling csv.c with an existing sqlite3 installation. For sqlite3 installed with Homebrew, it would be:
curl -O https://raw.githubusercontent.com/sqlite/sqlite/master/ext/misc/csv.c
gcc -g -I/usr/local/opt/sqlite/include -fPIC -dynamiclib csv.c -o csv.dylib
/usr/local/opt/sqlite/bin/sqlite3 '' '.load csv'
The last line is testing the newly compiled library in the Homebrew sqlite3. The default $ sqlite3 comes from the MacOS bundle and is outdated sometimes.

Answer 2

I managed to find a workaround and use an Ubuntu docker container, where I was able to successfully compile and use the extension. Here are the steps I followed:
docker run -ti --rm -v $(pwd):/host ubuntu bash
# Steps to build SQLite and CSV Virtual Table extension from source inside an Ubuntu docker container
cd /host
apt update
apt install -y vim build-essential zip wget
wget https://sqlite.org/2020/sqlite-autoconf-3310100.tar.gz
tar xvf sqlite-autoconf-3310100.tar.gz
cd sqlite-autoconf-3310100
./configure
make
./sqlite3
# Obtain csv.c and change <sqlite3ext.h> to "sqlite3ext.h" in the file
vi csv.c
gcc -g -fPIC -shared csv.c -o csv.so
./sqlite3
#sqlite> .load ./csv
#sqlite> CREATE VIRTUAL TABLE temp.t1 USING csv(filename='/host/users.csv',header);
#sqlite> .headers on
#sqlite> SELECT * FROM t1 LIMIT 1;
You can use a similar process to build for a Lambda function (CentOS). See this github repo for more details.
I still haven't figured out how to get it to compile directly on Mac, so any help would be greatly appreciated!

Answer 3

Leveraging off of Anton's answer for an Ubuntu-like distro, the following is mostly workable:
wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
mkdir bld
cd bld
../sqlite/configure
make
A gotchas in getting this far:
You may need to do this:
sudo apt-get install tcl8.6-dev
It will complain about not having access to tcl. If that's the case, you need to install and go back to the configure step.
Now, make should work unless you have other missing dependencies.
Assuming that works, now for the gcc:
gcc -I. -fPIC -shared ../sqlite/ext/misc/csv.c -o csv.so -lm
Now it's time to run. You need to set LD_LIBRARY_PATH. Assuming you are still in the bld directory:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`pwd`
or substitute the `pwd` for your full path to the csv.so file.
NOW! Now, you can run it. Enjoy.