I am trying to use the CSV Virtual Table extension for sqlite3. I get stuck on the first step of compiling the extension on a Mac (MacOS High Sierra 10.13.6).
I downloaded the source code for csv.c from this page. I also grabbed the sqlite source code amalgamation from here.
I used the following command to compile:
gcc -g -fPIC -dynamiclib csv.c -o csv.dylib
However, I get the following error:
csv.c:115:3: error: no member named '__builtin___vsnprintf_chk' in 'struct sqlite3_api_routines'
sqlite3_vsnprintf(CSV_MXERR, p->zErr, zFormat, ap);
^~~~~~~~~~~~~~~~~
/usr/include/sqlite3ext.h:437:53: note: expanded from macro 'sqlite3_vsnprintf'
#define sqlite3_vsnprintf sqlite3_api->vsnprintf
~~~~~~~~~~~ ^
/usr/include/secure/_stdio.h:75:3: note: expanded from macro 'vsnprintf'
__builtin___vsnprintf_chk (str, len, 0, __darwin_obsz(str), format, ap)
^
csv.c:115:21: warning: incompatible integer to pointer conversion passing 'int' to parameter of type 'const void *' [-Wint-conversion]
sqlite3_vsnprintf(CSV_MXERR, p->zErr, zFormat, ap);
^~~~~~~~~
csv.c:67:19: note: expanded from macro 'CSV_MXERR'
#define CSV_MXERR 200
^~~
/usr/include/secure/_stdio.h:75:57: note: expanded from macro 'vsnprintf'
__builtin___vsnprintf_chk (str, len, 0, __darwin_obsz(str), format, ap)
^~~
/usr/include/secure/_common.h:39:54: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^~~~~~
csv.c:568:5: error: use of undeclared identifier 'sqlite3_str'
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:568:18: error: use of undeclared identifier 'pStr'
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:568:25: warning: implicit declaration of function 'sqlite3_str_new' is invalid in C99 [-Wimplicit-function-declaration]
sqlite3_str *pStr = sqlite3_str_new(0);
^
csv.c:571:5: warning: implicit declaration of function 'sqlite3_str_appendf' is invalid in C99 [-Wimplicit-function-declaration]
sqlite3_str_appendf(pStr, "CREATE TABLE x(");
^
csv.c:571:25: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, "CREATE TABLE x(");
^
csv.c:581:29: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, "%sc%d TEXT", zSep, iCol);
^
csv.c:588:31: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr,"%s\"%w\" TEXT", zSep, z);
^
csv.c:597:31: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr,"%sc%d TEXT", zSep, ++iCol);
^
csv.c:603:25: error: use of undeclared identifier 'pStr'
sqlite3_str_appendf(pStr, ")");
^
csv.c:604:18: warning: implicit declaration of function 'sqlite3_str_finish' is invalid in C99 [-Wimplicit-function-declaration]
CSV_SCHEMA = sqlite3_str_finish(pStr);
^
csv.c:604:37: error: use of undeclared identifier 'pStr'
CSV_SCHEMA = sqlite3_str_finish(pStr);
^
csv.c:643:27: error: use of undeclared identifier 'SQLITE_VTAB_DIRECTONLY'
sqlite3_vtab_config(db, SQLITE_VTAB_DIRECTONLY);
^
4 warnings and 10 errors generated.
What am I doing wrong?
Richard Hipp at sqlite.org posted a solution on how to compile an SQLite extension.
The script for compiling the CSV extension looks like this (based on https://www.sqlite.org/loadext.html and https://github.com/sqlite/sqlite/blob/master/README.md):
wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
mkdir bld
cd bld
../sqlite/configure
make
gcc -g -I. -fPIC -dynamiclib ../sqlite/ext/misc/csv.c -o csv.dylib
Testing the script:
echo -e 'col_text,col_int\napples,3\noranges,5' > sample.csv
./sqlite3 '' '.load csv' 'CREATE VIRTUAL TABLE temp.t1 USING csv(filename="sample.csv");' 'SELECT * FROM t1;'
That's it.
You can also try compiling csv.c with an existing sqlite3 installation. For sqlite3 installed with Homebrew, it would be:
curl -O https://raw.githubusercontent.com/sqlite/sqlite/master/ext/misc/csv.c
gcc -g -I/usr/local/opt/sqlite/include -fPIC -dynamiclib csv.c -o csv.dylib
/usr/local/opt/sqlite/bin/sqlite3 '' '.load csv'
The last line is testing the newly compiled library in the Homebrew sqlite3. The default $ sqlite3 comes from the MacOS bundle and is outdated sometimes.
I managed to find a workaround and use an Ubuntu docker container, where I was able to successfully compile and use the extension. Here are the steps I followed:
docker run -ti --rm -v $(pwd):/host ubuntu bash
# Steps to build SQLite and CSV Virtual Table extension from source inside an Ubuntu docker container
cd /host
apt update
apt install -y vim build-essential zip wget
wget https://sqlite.org/2020/sqlite-autoconf-3310100.tar.gz
tar xvf sqlite-autoconf-3310100.tar.gz
cd sqlite-autoconf-3310100
./configure
make
./sqlite3
# Obtain csv.c and change <sqlite3ext.h> to "sqlite3ext.h" in the file
vi csv.c
gcc -g -fPIC -shared csv.c -o csv.so
./sqlite3
#sqlite> .load ./csv
#sqlite> CREATE VIRTUAL TABLE temp.t1 USING csv(filename='/host/users.csv',header);
#sqlite> .headers on
#sqlite> SELECT * FROM t1 LIMIT 1;
You can use a similar process to build for a Lambda function (CentOS). See this github repo for more details.
I still haven't figured out how to get it to compile directly on Mac, so any help would be greatly appreciated!
Leveraging off of Anton's answer for an Ubuntu-like distro, the following is mostly workable:
wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
mkdir bld
cd bld
../sqlite/configure
make
A gotchas in getting this far:
You may need to do this:
sudo apt-get install tcl8.6-dev
It will complain about not having access to tcl. If that's the case, you need to install and go back to the configure step.
Now, make should work unless you have other missing dependencies.
Assuming that works, now for the gcc:
gcc -I. -fPIC -shared ../sqlite/ext/misc/csv.c -o csv.so -lm
Now it's time to run. You need to set LD_LIBRARY_PATH. Assuming you are still in the bld directory:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`pwd`
or substitute the `pwd` for your full path to the csv.so file.
NOW! Now, you can run it. Enjoy.
I am trying compile a pexe file using the portable native client. I am following the instructions at this page:
https://developer.chrome.com/native-client/devguide/devcycle/building
I am trying to build a pexe and link it with a static lib as shown on the page. My command line is as follows.
/home/adev/opt/nacl_sdk/pepper_37/toolchain/linux_pnacl/bin/pnacl-clang++ -o pim-pnacl.pexe pim-pnacl.o -L/home/adev/opt/nacl_sdk/pepper_37/lib/pnacl/Release -lppapi_cpp -lppapi -L../libpidec -lpidec
and I get the following errors.
/home/adev/opt/nacl_sdk/pepper_37/toolchain/linux_pnacl/sdk/lib/libppapi_stub.a: error: undefined reference to 'PPP_GetInterface'
/home/adev/opt/nacl_sdk/pepper_37/toolchain/linux_pnacl/sdk/lib/libppapi_stub.a: error: undefined reference to 'PPP_ShutdownModule'
/home/adev/opt/nacl_sdk/pepper_37/toolchain/linux_pnacl/sdk/lib/libppapi_stub.a: error: undefined reference to 'PPP_InitializeModule'
For the record, the problem was in the pim-pnacl.o it needed to be compiled with the -c flag. This is the correct way to compile a single .o file:
/home/adev/opt/nacl_sdk/pepper_37/toolchain/linux_pnacl/bin/pnacl-clang++ -o pim-pnacl.o pim-pnacl.cc -O2 -I/home/adev/opt/nacl_sdk/pepper_37/include -c
I had omitted the -c and this caused linker problems.
I started a bisect qtwebkit and when I try to compile it in the first stage it gave me the following exception. I tried using a older qmake but still it gives the same exception. What does this really mean and how can I fix it.
g++ -c -Wall -Wreturn-type -fno-strict-aliasing -Wcast-align -Wchar-subscripts -Wformat-security -Wreturn-type -Wno-unused-parameter -Wno-sign-compare -Wno-switch -Wno-switch-enum -Wundef -Wmissing-noreturn -Winit-self -frtti -fexceptions -mthreads -DUNICODE -DQT_LARGEFILE_SUPPORT -DBUILDING_QT__=1 -DWTF_USE_ACCELERATED_COMPOSITING -DUSE_SYSTEM_MALLOC -DNDEBUG -D_HAS_TR1=0 -DBUILDING_QT__ -DBUILDING_JavaScriptCore -DBUILDING_WTF -DQT_CORE_LIB -DQT_HAVE_MMX -DQT_HAVE_3DNOW -DQT_HAVE_SSE -DQT_HAVE_MMXEXT -DQT_HAVE_SSE2 -DQT_THREAD_SUPPORT -I"..\..\..\..\include\QtCore" -I"..\..\..\..\include" -I"." -I"..\..\webkit" -I"assembler" -I"bytecode" -I"bytecompiler" -I"debugger" -I"interpreter" -I"jit" -I"parser" -I"pcre" -I"profiler" -I"runtime" -I"wtf" -I"wtf\symbian" -I"wtf\unicode" -I"yarr" -I"API" -I"ForwardingHeaders" -I"generated" -I"\include\QtWebKit" -I"pcre" -I"\JavaScriptCore\tmp" -I"..\..\..\..\include\ActiveQt" -I"tmp\moc\debug_shared" -I"c:\QtSDK1.2\mingw\include" -I"c:\QtSDK1.2\mingw\include" -I"..\..\..\..\mkspecs\win32-g++" -o tmp\obj\debug_shared\MainThreadQt.o wtf\qt\MainThreadQt.cpp In file included from wtf\qt\MainThreadQt.cpp:72:
tmp\moc\debug_shared/MainThreadQt.moc:13:2: error: #error "This file was generated using the moc from 4.8.0. It"
tmp\moc\debug_shared/MainThreadQt.moc:14:2: error: #error "cannot be used with the include files from this version of Qt."
tmp\moc\debug_shared/MainThreadQt.moc:15:2: error: #error "(The moc has changed too much.)"
In file included from wtf\qt\MainThreadQt.cpp:72:
tmp\moc\debug_shared/MainThreadQt.moc:42: error: no 'void WTF::MainThreadInvoker::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)' member function declared in class 'WTF::MainThreadInvoker'
tmp\moc\debug_shared/MainThreadQt.moc:55: error: 'const QMetaObjectExtraData WTF::MainThreadInvoker::staticMetaObjectExtraData' is not a static member of 'class WTF::MainThreadInvoker'
tmp\moc\debug_shared/MainThreadQt.moc:56: error: 'qt_static_metacall' was not declared in this scope
tmp\moc\debug_shared/MainThreadQt.moc:57: error: too many initializers for 'const QMetaObjectExtraData'
tmp\moc\debug_shared/MainThreadQt.moc:61: error: 'staticMetaObjectExtraData' was not declared in this scope
tmp\moc\debug_shared/MainThreadQt.moc:62: error: too many initializers for 'QMetaObject::<anonymous struct>'
tmp\moc\debug_shared/MainThreadQt.moc: In member function 'virtual int WTF::MainThreadInvoker::qt_metacall(QMetaObject::Call, int, void**)':
tmp\moc\debug_shared/MainThreadQt.moc:88: error: 'qt_static_metacall' was not declared in this scope
mingw32-make[2]: *** [tmp/obj/debug_shared/MainThreadQt.o] Error 1
mingw32-make[2]: Leaving directory `C:/Qt/qtGITSource/qt/src/3rdparty/webkit/JavaScriptCore'
mingw32-make[1]: *** [debug-all] Error 2
mingw32-make[1]: Leaving directory `C:/Qt/qtGITSource/qt/src/3rdparty/webkit/JavaScriptCore'
mingw32-make: *** [sub-JavaScriptCore-make_default-ordered] Error 2
IMHO ll. 2,3 is important:
error "This file was generated using the moc from 4.8.0. It"
error "cannot be used with the include files from this version of Qt."
Your reach a version less than qt 4.8 and all your moc-files were generated in a previous build by moc v4.8. You have to clean your build directory and restart your build command (ensure you use a proper version of qmake, moc etc.).
I am trying to compile Qt embedded for ARM. I have followed the instructions both here and here. I am using latest Angstrom toolchain and qt-embedded-linux-opensource-src-4.5.3 and I am using Ubuntu 10.10 x86.
Here steps I past:
Configured qws/linux-arm-g++/qmake.conf file for my toolchain. Here it's content:
#
# qmake configuration for building with arm-linux-g++
#
include(../../common/g++.conf)
include(../../common/linux.conf)
include(../../common/qws.conf)
# modifications to g++.conf
QMAKE_CC = arm-angstrom-linux-gnueabi-gcc
QMAKE_CXX = arm-angstrom-linux-gnueabi-g++
QMAKE_LINK = arm-angstrom-linux-gnueabi-g++
QMAKE_LINK_SHLIB = arm-angstrom-linux-gnueabi-g++
# modifications to linux.conf
QMAKE_AR = arm-angstrom-linux-gnueabi-ar cqs
QMAKE_OBJCOPY = arm-angstrom-linux-gnueabi-objcopy
QMAKE_STRIP = arm-angstrom-linux-gnueabi-strip
load(qt_config)
Set my toolchain path:
export PATH=/home/olcay/angstrom/arm/bin:$PATH
Configured qt with:
./configure -arch arm -little-endian -xplatform qws/linux-arm-g++
Run make.
While running make I am getting this error:
make[1]: Entering directory `/home/olcay/qt-embedded/src/corelib'
arm-angstrom-linux-gnueabi-g++ -c -pipe -fno-exceptions -O2 -Wall -W -D_REENTRANT -fPIC -DQT_SHARED -DQT_BUILD_CORE_LIB -DQT_NO_USING_NAMESPACE -DQT_NO_CAST_TO_ASCII -DQT_ASCII_CAST_WARNINGS -DQT3_SUPPORT -DQT_MOC_COMPAT -DHB_EXPORT=Q_CORE_EXPORT -DQT_NO_DEBUG -I../../mkspecs/qws/linux-arm-g++ -I. -I../../include -I../../include/QtCore -I.rcc/release-shared-emb-arm -Iglobal -I../3rdparty/zlib -I../3rdparty/harfbuzz/src -I.moc/release-shared-emb-arm -o .obj/release-shared-emb-arm/qatomic_arm.o arch/arm/qatomic_arm.cpp
/home/olcay/angstrom/arm/bin/arm-angstrom-linux-gnueabi-g++: 1: ELF: not found
/home/olcay/angstrom/arm/bin/arm-angstrom-linux-gnueabi-g++: 2: Syntax error: word unexpected (expecting ")")
make[1]: *** [.obj/release-shared-emb-arm/qatomic_arm.o] Error 2
make[1]: Leaving directory `/home/olcay/qt-embedded/src/corelib'
make: *** [sub-corelib-make_default-ordered] Error 2
I have also noticed that while config there is a warning and same error line in make:
The system floating point format could not be detected.
This may cause data to be generated in a wrong format
Turn on verbose messaging (-v) to see the final report.
/home/olcay/angstrom/arm/bin/arm-angstrom-linux-gnueabi-g++: 1: ELF: not found
/home/olcay/angstrom/arm/bin/arm-angstrom-linux-gnueabi-g++: 2: Syntax error: word unexpected (expecting ")")
And changing configure line as #Luca Carlon described does not solve the problem.
Please help me!
Thanks in advance.
I have noticed that I have downloaded my Angstrom tool-chain for 64 bit host platform but I am using 32 bit host. After downloading correct tool-chain I have managed to compile Qt. And I found that an "ELF not found" error occurs when you try to run an application which is not build for your architecture.
Replace -arch arm with -embedded arm and try again if you want to build Qt Embedded.
I'm trying to use QtService from QtSolution in a Qt project.
I've unzipped it and run:
configure.bat -library
qmake
mingw32-make
the debug and release dll were placed inside C:\Qt\2010.03\qt\bin\
Examples were correctly built .
I added this line at the bottom of my .pro file:
include(../QtSolutions/qtservice/src/qtservice.pri)
and I was able to browse the qtservice's sources from the project source tree.
When I try to build my project I get:
Running build steps for project EMCSolutionCrawler...
Configuration unchanged, skipping QMake step.
Starting: C:/Qt/2010.03/mingw/bin/mingw32-make.exe -w
mingw32-make: Entering directory `C:/Progetti/EMCSolutionCrawler'
c:\qt\2010.03\qt\bin\qmake.exe -spec ..\..\Qt\2010.03\qt\mkspecs\win32-g++ -win32 -o Makefile EMCSolutionCrawler.pro
mingw32-make: Leaving directory `C:/Progetti/EMCSolutionCrawler'
mingw32-make: Entering directory `C:/Progetti/EMCSolutionCrawler'
C:/Qt/2010.03/mingw/bin/mingw32-make -f Makefile.Debug
mingw32-make[1]: Entering directory `C:/Progetti/EMCSolutionCrawler'
g++ -g -frtti -fexceptions -mthreads -Wall -DUNICODE -DQT_LARGEFILE_SUPPORT -DQT_QTSERVICE_IMPORT -DQT_DLL -DQT_SQL_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_THREAD_SUPPORT ../QtSolutions/qtservice/src/QtService.cpp -o ../QtSolutions/qtservice/src/QtService
mingw32-make[1]: Leaving directory `C:/Progetti/EMCSolutionCrawler'
mingw32-make: Leaving directory `C:/Progetti/EMCSolutionCrawler'
In file included from ../QtSolutions/qtservice/src/QtService.cpp:47:
../QtSolutions/qtservice/src/qtservice.h:50:35: error: QtCore/QCoreApplication: No such file or directory
In file included from ../QtSolutions/qtservice/src/QtService.cpp:48:
../QtSolutions/qtservice/src/qtservice_p.h:50:30: error: QtCore/QStringList: No such file or directory
../QtSolutions/qtservice/src/QtService.cpp:51:25: error: QtCore/QTimer: No such file or directory
../QtSolutions/qtservice/src/QtService.cpp:52:26: error: QtCore/QVector: No such file or directory
../QtSolutions/qtservice/src/QtService.cpp:53:27: error: QtCore/QProcess: No such file or directory
../QtSolutions/qtservice/src/QtService.cpp:426:25: error: qtservice.moc: No such file or directory
In file included from ../QtSolutions/qtservice/src/QtService.cpp:47:
../QtSolutions/qtservice/src/qtservice.h:73: error: ISO C++ forbids declaration of 'Q_DECLARE_PRIVATE' with no type
../QtSolutions/qtservice/src/qtservice.h:74: error: expected ';' before 'public'
../QtSolutions/qtservice/src/qtservice.h:80: error: ISO C++ forbids declaration of 'QString' with no type
../QtSolutions/qtservice/src/qtservice.h:80: error: expected ',' or '...' before '&' token
../QtSolutions/qtservice/src/qtservice.h:86: error: 'QString' does not name a type
../QtSolutions/qtservice/src/qtservice.h:87: error: 'QString' does not name a type
../QtSolutions/qtservice/src/qtservice.h:88: error: 'StartupType' does not name a type
../QtSolutions/qtservice/src/qtservice.h:89: error: 'QString' does not name a type
../QtSolutions/qtservice/src/qtservice.h:91: error: ISO C++ forbids declaration of 'QString' with no type
../QtSolutions/qtservice/src/qtservice.h:91: error: expected ',' or '...' before '&' token
../QtSolutions/qtservice/src/qtservice.h:110: error: ISO C++ forbids declaration of 'Q_DECLARE_PRIVATE' with no type
../QtSolutions/qtservice/src/qtservice.h:111: error: expected ';' before 'public'
../QtSolutions/qtservice/src/qtservice.h:125: error: 'ServiceFlags' has not been declared
../QtSolutions/qtservice/src/qtservice.h:125: error: ISO C++ forbids declaration of 'Q_DECLARE_FLAGS' with no type
../QtSolutions/qtservice/src/qtservice.h:127: error: expected ';' before 'QtServiceBase'
../QtSolutions/qtservice/src/qtservice.h:130: error: 'QString' does not name a type
...
What am I missing?
Environment infomation:
Qt 4.6.3 (mingw - LGPL)
Windows XP SP3
QtService 2.6
Thank you for your time.
Just a guess: I don't see a include -IC:\Qt\2010.03\qt\include in your call of g++. So the compiler can't find the Qt Headers. I tink there is something wrong in your EMCSolutionCrawler.pro.