I am trying to use the CSV Virtual Table extension for sqlite3. I get stuck on the first step of compiling the extension on a Mac (MacOS High Sierra 10.13.6).
I downloaded the source code for csv.c from this page. I also grabbed the sqlite source code amalgamation from here.
I used the following command to compile:
gcc -g -fPIC -dynamiclib csv.c -o csv.dylib
However, I get the following error:
csv.c:115:3: error: no member named '__builtin___vsnprintf_chk' in 'struct sqlite3_api_routines'
sqlite3_vsnprintf(CSV_MXERR, p->zErr, zFormat, ap);
^~~~~~~~~~~~~~~~~
/usr/include/sqlite3ext.h:437:53: note: expanded from macro 'sqlite3_vsnprintf'
#define sqlite3_vsnprintf sqlite3_api->vsnprintf
~~~~~~~~~~~ ^
/usr/include/secure/_stdio.h:75:3: note: expanded from macro 'vsnprintf'
__builtin___vsnprintf_chk (str, len, 0, __darwin_obsz(str), format, ap)
^
csv.c:115:21: warning: incompatible integer to pointer conversion passing 'int' to parameter of type 'const void *' [-Wint-conversion]
sqlite3_vsnprintf(CSV_MXERR, p->zErr, zFormat, ap);
^~~~~~~~~
csv.c:67:19: note: expanded from macro 'CSV_MXERR'
#define CSV_MXERR 200
^~~
/usr/include/secure/_stdio.h:75:57: note: expanded from macro 'vsnprintf'
__builtin___vsnprintf_chk (str, len, 0, __darwin_obsz(str), format, ap)
^~~
/usr/include/secure/_common.h:39:54: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^~~~~~
csv.c:568:5: error: use of undeclared identifier 'sqlite3_str'
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:568:18: error: use of undeclared identifier 'pStr'
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:568:25: warning: implicit declaration of function 'sqlite3_str_new' is invalid in C99 [-Wimplicit-function-declaration]
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:571:5: warning: implicit declaration of function 'sqlite3_str_appendf' is invalid in C99 [-Wimplicit-function-declaration]
sqlite3_str_appendf(pStr, "CREATE TABLE x(");
^
csv.c:571:25: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, "CREATE TABLE x(");
^
csv.c:581:29: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, "%sc%d TEXT", zSep, iCol);
^
csv.c:588:31: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr,"%s\"%w\" TEXT", zSep, z);
^
csv.c:597:31: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr,"%sc%d TEXT", zSep, ++iCol);
^
csv.c:603:25: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, ")");
^
csv.c:604:18: warning: implicit declaration of function 'sqlite3_str_finish' is invalid in C99 [-Wimplicit-function-declaration]
CSV_SCHEMA = sqlite3_str_finish(pStr);
^
csv.c:604:37: error: use of undeclared identifier 'pStr'
CSV_SCHEMA = sqlite3_str_finish(pStr);
^
csv.c:643:27: error: use of undeclared identifier 'SQLITE_VTAB_DIRECTONLY'
sqlite3_vtab_config(db, SQLITE_VTAB_DIRECTONLY);
^
4 warnings and 10 errors generated.
What am I doing wrong?
Richard Hipp at sqlite.org posted a solution on how to compile an SQLite extension.
The script for compiling the CSV extension looks like this (based on https://www.sqlite.org/loadext.html and https://github.com/sqlite/sqlite/blob/master/README.md):
wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
mkdir bld
cd bld
../sqlite/configure
make
gcc -g -I. -fPIC -dynamiclib ../sqlite/ext/misc/csv.c -o csv.dylib
Testing the script:
echo -e 'col_text,col_int\napples,3\noranges,5' > sample.csv
./sqlite3 '' '.load csv' 'CREATE VIRTUAL TABLE temp.t1 USING csv(filename="sample.csv");' 'SELECT * FROM t1;'
That's it.
You can also try compiling csv.c with an existing sqlite3 installation. For sqlite3 installed with Homebrew, it would be:
curl -O https://raw.githubusercontent.com/sqlite/sqlite/master/ext/misc/csv.c
gcc -g -I/usr/local/opt/sqlite/include -fPIC -dynamiclib csv.c -o csv.dylib
/usr/local/opt/sqlite/bin/sqlite3 '' '.load csv'
The last line is testing the newly compiled library in the Homebrew sqlite3. The default $ sqlite3 comes from the MacOS bundle and is outdated sometimes.
I managed to find a workaround and use an Ubuntu docker container, where I was able to successfully compile and use the extension. Here are the steps I followed:
docker run -ti --rm -v $(pwd):/host ubuntu bash
# Steps to build SQLite and CSV Virtual Table extension from source inside an Ubuntu docker container
cd /host
apt update
apt install -y vim build-essential zip wget
wget https://sqlite.org/2020/sqlite-autoconf-3310100.tar.gz
tar xvf sqlite-autoconf-3310100.tar.gz
cd sqlite-autoconf-3310100
./configure
make
./sqlite3
# Obtain csv.c and change <sqlite3ext.h> to "sqlite3ext.h" in the file
vi csv.c
gcc -g -fPIC -shared csv.c -o csv.so
./sqlite3
#sqlite> .load ./csv
#sqlite> CREATE VIRTUAL TABLE temp.t1 USING csv(filename='/host/users.csv',header);
#sqlite> .headers on
#sqlite> SELECT * FROM t1 LIMIT 1;
You can use a similar process to build for a Lambda function (CentOS). See this github repo for more details.
I still haven't figured out how to get it to compile directly on Mac, so any help would be greatly appreciated!
Leveraging off of Anton's answer for an Ubuntu-like distro, the following is mostly workable:
wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
mkdir bld
cd bld
../sqlite/configure
make
A gotchas in getting this far:
You may need to do this:
sudo apt-get install tcl8.6-dev
It will complain about not having access to tcl. If that's the case, you need to install and go back to the configure step.
Now, make should work unless you have other missing dependencies.
Assuming that works, now for the gcc:
gcc -I. -fPIC -shared ../sqlite/ext/misc/csv.c -o csv.so -lm
Now it's time to run. You need to set LD_LIBRARY_PATH. Assuming you are still in the bld directory:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`pwd`
or substitute the `pwd` for your full path to the csv.so file.
NOW! Now, you can run it. Enjoy.
I am using QT Binding/Wrapper for Go in my project. Project is compiling fine for amd64 but when I try to compile project for arm devices, it gives me below error
github.com/therecipe/qt/core
# github.com/therecipe/qt/core
g++: error: unrecognized command line option ‘-marm’
Please find below my go environment variables setup
GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/minhaj/GoLang"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
GOARM=""
CC="arm-linux-gnueabi-gcc"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-
map=/tmp/go-build277718108=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
I'm trying to get Google native client to work on windows and I follow their steps. I installed pepper and while I'm in the getting started folder I type make serve but I get a make (e=5): Access is denied error...
Here is the whole error message :
C:\Users\User123\PROJET C\NaCl\nacl_sdk\pepper_35\getting_started>make serve
C:/Users/User123/PROJET C/NaCl/nacl_sdk/pepper_35/tools/make -C part1 all
C:\Python27\python.exe: can't find '__main__' module in 'C:/Users/User123'
make[1]: Entering directory 'C:/Users/User123/PROJET C/NaCl/nacl_sdk/pepper_35/gett
ing_started/part1'
C:/Users/User123 C:/Users/User123/PROJET C:/Users/User123/PROJET C/NaCl/nacl_sdk/pepper_3
5/getting_started/part1/C/NaCl/nacl_sdk/pepper_35/getting_started/part1/C/NaCl/n
acl_sdk/pepper_35/toolchain/_pnacl/bin/pnacl-clang++ -o hello_tutorial.bc hello_
tutorial.cc -O2 -IC:/Users/User123 C:/Users/User123/PROJET C/NaCl/nacl_sdk/pepper_35/g
etting_started/part1/C/NaCl/nacl_sdk/pepper_35/include -LC:/Users/User123 C:/Users/
User123/PROJET C/NaCl/nacl_sdk/pepper_35/getting_started/part1/C/NaCl/nacl_sdk/pepp
er_35/lib/pnacl/Release -lppapi_cpp -lppapi
process_begin: CreateProcess(NULL, C:/Users/User123 C:/Users/User123/PROJET C:/Users/V
olt/PROJET C/NaCl/nacl_sdk/pepper_35/getting_started/part1/C/NaCl/nacl_sdk/peppe
r_35/getting_started/part1/C/NaCl/nacl_sdk/pepper_35/toolchain/_pnacl/bin/pnacl-
clang++ -o hello_tutorial.bc hello_tutorial.cc -O2 -IC:/Users/User123 C:/Users/User123
/PROJET C/NaCl/nacl_sdk/pepper_35/getting_started/part1/C/NaCl/nacl_sdk/pepper_3
5/include -LC:/Users/User123 C:/Users/User123/PROJET C/NaCl/nacl_sdk/pepper_35/getting
_started/part1/C/NaCl/nacl_sdk/pepper_35/lib/pnacl/Release -lppapi_cpp -lppapi,
...) failed.
make (e=5): Access is denied.
Makefile:50: recipe for target 'hello_tutorial.bc' failed
make[1]: *** [hello_tutorial.bc] Error 5
make[1]: Leaving directory 'C:/Users/User123/PROJET C/NaCl/nacl_sdk/pepper_35/getti
ng_started/part1'
Makefile:46: recipe for target 'part1_ALL_TARGET' failed
make: *** [part1_ALL_TARGET] Error 2
C:\Users\User123\PROJET C\NaCl\nacl_sdk\pepper_35\getting_started>
How could I fix this ? I dont think it comes from files rights...
This looks like its related to the space you have in the name of your folder. Can you try removing the space?
I tried following the tutorial here for NaCl or PNaCl or native client or pinnacle whatever the name of the day is and got this:
F:/nacl_sdk/pepper_31/toolchain/win_x86_newlib/bin/i686-nacl-g++ -o hello_tutori
al_x86_32.nexe hello_tutorial_32.o -m32 -O0 -g -IF:/nacl_sdk/pepper_31/include -
lppapi_cpp -lppapi
/x86_64-nacl-ld: cannot find -lppapi_cpp
collect2: ld returned 1 exit status
Makefile:64: recipe for target 'hello_tutorial_x86_32.nexe' failed
make: * [hello_tutorial_x86_32.nexe] Error 1
Any ideas?
It looks like you are running the old "hello_tutorial" example with the new pepper_31 SDK. Please use this new documentation instead:
https://developers.google.com/native-client/dev/devguide/tutorial/tutorial-part1
This source for this tutorial can be found in the pepper_31 SDK in the directory pepper_31\getting_started\part1.
The new tutorial cited by binji is set up to build/run a pexe. To use nexe's as in the older example, update the tutorial's Makefile:
Add LDFLAGS for each target:
LDFLAGS_NEWLIB_ARM := -L$(NACL_SDK_ROOT)/lib/newlib_arm/Release -lppapi_cpp -lppapi
LDFLAGS_NEWLIB_X86_32 := -L$(NACL_SDK_ROOT)/lib/newlib_x86_32/Release -lppapi_cpp -lppapi
LDFLAGS_NEWLIB_X86_64 := -L$(NACL_SDK_ROOT)/lib/newlib_x86_64/Release -lppapi_cpp -lppapi
and change the $(LDFLAGS) specified for linking the respective targets.
I started a bisect qtwebkit and when I try to compile it in the first stage it gave me the following exception. I tried using a older qmake but still it gives the same exception. What does this really mean and how can I fix it.
g++ -c -Wall -Wreturn-type -fno-strict-aliasing -Wcast-align -Wchar-subscripts -Wformat-security -Wreturn-type -Wno-unused-parameter -Wno-sign-compare -Wno-switch -Wno-switch-enum -Wundef -Wmissing-noreturn -Winit-self -frtti -fexceptions -mthreads -DUNICODE -DQT_LARGEFILE_SUPPORT -DBUILDING_QT__=1 -DWTF_USE_ACCELERATED_COMPOSITING -DUSE_SYSTEM_MALLOC -DNDEBUG -D_HAS_TR1=0 -DBUILDING_QT__ -DBUILDING_JavaScriptCore -DBUILDING_WTF -DQT_CORE_LIB -DQT_HAVE_MMX -DQT_HAVE_3DNOW -DQT_HAVE_SSE -DQT_HAVE_MMXEXT -DQT_HAVE_SSE2 -DQT_THREAD_SUPPORT -I"..\..\..\..\include\QtCore" -I"..\..\..\..\include" -I"." -I"..\..\webkit" -I"assembler" -I"bytecode" -I"bytecompiler" -I"debugger" -I"interpreter" -I"jit" -I"parser" -I"pcre" -I"profiler" -I"runtime" -I"wtf" -I"wtf\symbian" -I"wtf\unicode" -I"yarr" -I"API" -I"ForwardingHeaders" -I"generated" -I"\include\QtWebKit" -I"pcre" -I"\JavaScriptCore\tmp" -I"..\..\..\..\include\ActiveQt" -I"tmp\moc\debug_shared" -I"c:\QtSDK1.2\mingw\include" -I"c:\QtSDK1.2\mingw\include" -I"..\..\..\..\mkspecs\win32-g++" -o tmp\obj\debug_shared\MainThreadQt.o wtf\qt\MainThreadQt.cpp In file included from wtf\qt\MainThreadQt.cpp:72:
tmp\moc\debug_shared/MainThreadQt.moc:13:2: error: #error "This file was generated using the moc from 4.8.0. It"
tmp\moc\debug_shared/MainThreadQt.moc:14:2: error: #error "cannot be used with the include files from this version of Qt."
tmp\moc\debug_shared/MainThreadQt.moc:15:2: error: #error "(The moc has changed too much.)"
In file included from wtf\qt\MainThreadQt.cpp:72:
tmp\moc\debug_shared/MainThreadQt.moc:42: error: no 'void WTF::MainThreadInvoker::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)' member function declared in class 'WTF::MainThreadInvoker'
tmp\moc\debug_shared/MainThreadQt.moc:55: error: 'const QMetaObjectExtraData WTF::MainThreadInvoker::staticMetaObjectExtraData' is not a static member of 'class WTF::MainThreadInvoker'
tmp\moc\debug_shared/MainThreadQt.moc:56: error: 'qt_static_metacall' was not declared in this scope
tmp\moc\debug_shared/MainThreadQt.moc:57: error: too many initializers for 'const QMetaObjectExtraData'
tmp\moc\debug_shared/MainThreadQt.moc:61: error: 'staticMetaObjectExtraData' was not declared in this scope
tmp\moc\debug_shared/MainThreadQt.moc:62: error: too many initializers for 'QMetaObject::<anonymous struct>'
tmp\moc\debug_shared/MainThreadQt.moc: In member function 'virtual int WTF::MainThreadInvoker::qt_metacall(QMetaObject::Call, int, void**)':
tmp\moc\debug_shared/MainThreadQt.moc:88: error: 'qt_static_metacall' was not declared in this scope
mingw32-make[2]: *** [tmp/obj/debug_shared/MainThreadQt.o] Error 1
mingw32-make[2]: Leaving directory `C:/Qt/qtGITSource/qt/src/3rdparty/webkit/JavaScriptCore'
mingw32-make[1]: *** [debug-all] Error 2
mingw32-make[1]: Leaving directory `C:/Qt/qtGITSource/qt/src/3rdparty/webkit/JavaScriptCore'
mingw32-make: *** [sub-JavaScriptCore-make_default-ordered] Error 2
IMHO ll. 2,3 is important:
error "This file was generated using the moc from 4.8.0. It"
error "cannot be used with the include files from this version of Qt."
Your reach a version less than qt 4.8 and all your moc-files were generated in a previous build by moc v4.8. You have to clean your build directory and restart your build command (ensure you use a proper version of qmake, moc etc.).