I'm trying to install python package airflow into a virtualenv that has been created using pipenv, inside a docker container. It fails with an error that I'm clueless about.
Here is my Dockerfile:
FROM python:3.6-stretch
WORKDIR /tmp
# Define build args
ARG http_proxy
ARG https_proxy
ARG no_proxy
RUN apt-get update && \
apt-get -y install default-jdk
# Detect JAVA_HOME and export in bashrc.
# This will result in something like this being added to /etc/bash.bashrc
# export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
RUN echo export JAVA_HOME="$(readlink -f /usr/bin/java | sed "s:/jre/bin/java::")" >> /etc/bash.bashrc
# Upgrade pip
RUN pip install --upgrade pip
# Install core python packages
RUN pip install pipenv==2018.5.18
Build and run:
docker build -t pipenvtest:latest .
docker run -it pipenvtest:latest bash
When connected to the container:
pipenv --python 2.7
pipenv install --dev airflow
Which fails with this error:
building '_yaml' extension
creating build/temp.linux-x86_64-2.7/ext
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c ext/_yaml.c -o build/temp.linux-x86_64-2.7/ext/_yaml.o
ext/_yaml.c:4:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
(the ^ actually appears at the end of the line preceding it but I don't know how to format the quoted text as such)
I admit to not having the faintest idea how to go about solving this so hoping someone can give me some pointers. I hope the repro that I've included here works for you.
Is the --dev switch in pipenv install --dev airflow intended? It instructs pipenv to install development dependencies of Airflow too. One of these dependencies needs the Python.h header file (which is missing). To resolve the problem:
If you do not need the development dependencies then
remove the --dev switch.
If you need the development
dependencies then install the libpython2.7-dev package, which
provides Pthon.h, before you install Aiflow: apt install libpython2.7-dev
OK, I was being really dumb. I was trying to setup a python2.7 virtualenv on an image built from python:3.6-stretch.
I changed
pipenv --python 2.7
to
pipenv --python 3.6
and it worked.
Related
I tried to use the rocksdb inside R package. I used the following src/Makevars:
CXX_STD = CXX11
PKG_CPPFLAGS = -I./rocksdb/include/
PKG_LIBS = rocksdb/librocksdb.a -lbz2 -lz -lzstd -llz4 -lsnappy
$(SHLIB): rocksdb/librocksdb.a
rocksdb/librocksdb.a: rocksdb/Makefile
CFLAGS="$(CFLAGS) $(CPICFLAGS)" AR="$(AR)" RANLIB="$(RANLIB)" LDFLAGS="$(LDFLAGS)" \
$(MAKE) -d --jobs=3 --directory=rocksdb static_lib
clean:
$(MAKE) --directory=rocksdb clean
Package installation failed with many errors (see build log below).
You could reproduce this case using Docker container:
Necessary commands:
docker run --rm -ti rocker/r-ver:latest bash
Execute in container:
apt-get update
# install system deps
apt-get install -y libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev
apt-get install -y git-core
# install R deps
install2.r Rcpp checkmate R6 tinytest
cd /tmp
git clone https://gitlab.com/artemklevtsov/rocksdb
cd rocksdb/
git submodule init
git submodule update
R CMD INSTALL .
But I can successfully run make directly in the rocksdb source directory:
cd src/rocksdb/
make static_lib
How can I fix src/Makevars to build rocksdb during R package installation?
Links:
build log
rocksdb install guide
R package source repo
Not a full answer but an observation (for now):
I tried to reproduce this in a docker container. The R package build failed, but also the plain build when using the same flags as used by R, but without parallel jobs and make's debug output:
root#e8749c4bca63:/tmp/rocksdb/src# CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -fpic" AR="ar" RANLIB="ranlib" LDFLAGS="-L/usr/local/lib" \
> make --directory=rocksdb static_lib
[...]
CC util/bloom.o
CC util/build_version.o
util/build_version.cc:5:42: error: macro "__DATE__" might prevent reproducible builds [-Werror=date-time]
const char* rocksdb_build_compile_date = __DATE__;
^~~~~~~~
cc1plus: all warnings being treated as errors
Makefile:2029: recipe for target 'util/build_version.o' failed
make: *** [util/build_version.o] Error 1
make: Leaving directory '/tmp/rocksdb/src/rocksdb'
So it looks as if -Wdate-time got promoted to -Werror=date-time.
To solve the problem we should reset MAKEFLAGS variable. So the right way to do it:
rocksdb/librocksdb.a: rocksdb/Makefile
CFLAGS="$(CCFLAGS) $(CPICFLAGS)" MAKEFLAGS="" \
$(MAKE) -C rocksdb DISABLE_WARNING_AS_ERROR=1 static_lib
MAKEFLAGS content:
MAKEFLAGS= -- OBJECTS=RcppExports.o\ backup.o\ checkpoint.o\ db.o\ del.o\ exists.o\ get.o\ keys.o\ list.o\ options.o\ property.o\ put.o\ size.o\ sst.o\ utils.o\ version.o\ wrap.o SHLIB=rocksdb.so SHLIB_LD=$$(SHLIB_CXX11LD) SHLIB_LDFLAGS=$$(SHLIB_CXX11LDFLAGS) CXXPICFLAGS=$$(CXX11PICFLAGS) CXXFLAGS=$$(CXX11FLAGS) CXX=$$(CXX11)\ $$(CXX11STD)
I want to study sqlite by reading its source code. I found sqlite-1.0.1 only contained 10,000+ lines code, maybe it is easy to read. But unfortunately, modern gcc cannot compile sqlite-1.0.1.
wget "https://www.sqlite.org/src/tarball/e8521fc1/SQLite-e8521fc1.tar.gz"
tar xzvf SQLite-e8521fc1.tar.gz
mkdir bld && cd bld
../SQLite-e8521fc1/configure --prefix=/opt/sqlite-1.0.1 --with-tcl=no
make
When I issued make, it reported
gcc -std=c89 -g -O2 -o lemon ../SQLite-e8521fc1/tool/lemon.c
In file included from ../SQLite-e8521fc1/tool/lemon.c:29:
/usr/lib/gcc/i386-redhat-linux/3.4.2/include/varargs.h:4:2: #error "GCC no longer implements <varargs.h>."
/usr/lib/gcc/i386-redhat-linux/3.4.2/include/varargs.h:5:2: #error "Revise your code to use <stdarg.h>."
I tried appending -std=c89 following gcc in Makefile, but it did not work.
I specially installed a old Fedora Core release 3 virtual machine to compile SQLite-1.0.1, but it looks like the gcc(gcc version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)) within Fedora 3 is still too modern for the 18 years ago code.
I found a check in which <varargs.h> was replaced by <stdarg.h>, and the author said maybe he has used gcc to compile lemon.c in 1989.
Maybe a very old gcc or Linux (before 2000) can solve this problem, but how can I get that one?
I have modified the code of sqlite 1.0.1 and can now compile and run successfully on fedora 36 & MacOS 12.3. You can find this code on github. It can be compiled in docker as follows.(Note: You can also use gcc instead of clang)
docker pull fedora:36
docker run -dt --name f36 --cap-add=SYS_PTRACE --security-opt seccomp=unconfined fedora:36
docker exec -it f36 bash
dnf update -y
dnf install -y ncurses git hostname clang diffutils readline-devel tcl-devel gdbm-devel
mkdir ~/src && cd ~/src
git clone https://github.com/ruomeng0x/sqlite.git
cd ~/src/sqlite
git checkout badcc1d968fdb7cf0ff5d89468248468b8329f4e
mkdir ../build && cd ../build
CC=clang CFLAGS="-std=c89" bash ../sqlite/configure
make
I've tried to solve this using the previous questions/answers on SO but without any success. So, here's my problem.
I'm using RStudio on and Ubuntu box (14.04) and I tried to upgrade rJava from sources and in the process I managed to lose it.
I tried to install it again using,
install.packages("rJava")
which returned the following error message,
configure: error: One or more Java configuration variables are not set.
Make sure R is configured with full Java support (including JDK). Run
R CMD javareconf
as root to add Java support to R.
If you don't have root privileges, run
R CMD javareconf -e
to set all Java-related variables and then install rJava.
ERROR: configuration failed for package ‘rJava’
* removing ‘/home/darren/R/x86_64-pc-linux-gnu-library/3.2/rJava’
Warning in install.packages :
installation of package ‘rJava’ had non-zero exit status
So, I went to the terminal and typed,
sudo R CMD javareconf
which also gave the following error,
trying to compile and link a JNI program
detected JNI cpp flags :
detected JNI linker flags : -L/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server -ljvm
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -fpic -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2 -g -c conftest.c -o conftest.o
conftest.c:1:17: fatal error: jni.h: No such file or directory
#include <jni.h>
^
compilation terminated.
make: *** [conftest.o] Error 1
Unable to compile a JNI program
JAVA_HOME : /usr/lib/jvm/default-java
Java library path:
JNI cpp flags :
JNI linker flags :
Updating Java configuration in /usr/lib/R
Done.
I tried to follow these links, one and two but they didn't seem to resolve my issue; there are more links on SO but I'm not sure which one to follow. I've also un-installed and re-installed RStudio via the Ubuntu Software Centre but this didn't make any difference.
Can anyone else help?
In short, I want to be able to use RStudio with rJava again without it destroying any other uses of Java (such as jmol).
You don't seem to have JDK installed. You will need at least
sudo apt-get install openjdk-7-jdk
then re-run
sudo R CMD javareconf
Make sure you do NOT set JAVA_HOME by hand - it will be detected automatically. You should then see something like this:
$ sudo R CMD javareconf
Java interpreter : /usr/bin/java
Java version : 1.7.0_91
Java home path : /usr/lib/jvm/java-7-openjdk-amd64/jre
Java compiler : /usr/bin/javac
Java headers gen.: /usr/bin/javah
Java archive tool: /usr/bin/jar
trying to compile and link a JNI program
detected JNI cpp flags : -I$(JAVA_HOME)/../include
detected JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/usr/lib/jvm/java-7-openjdk-amd64/jre/../include -fpic -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g -c conftest.c -o conftest.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o conftest.so conftest.o -L/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server -ljvm -L/usr/lib/R/lib -lR
JAVA_HOME : /usr/lib/jvm/java-7-openjdk-amd64/jre
Java library path: $(JAVA_HOME)/lib/amd64/server
JNI cpp flags : -I$(JAVA_HOME)/../include
JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
What is wrong with sudo apt-get install r-cran-rjava ?
See for example this earlier answer and the question / thread around it.
For an installation from scratch, you could still much worse than starting from sudo apt-get build-dep r-cran-rjava. It will get you the JDK corresponding to your Ubuntu version.
First i would recommend installing Rstudio from its website: https://www.rstudio.com/products/rstudio/download/ (i.e. Rstudio 64bit: https://download1.rstudio.org/rstudio-0.99.489-amd64.deb). This does not solve the problem directly, but it helps to avoid other bugs with Rstudio.
Regarding the error, trying to make sure you have JDK install. I don't think the command java -version can tell if JDK is installed. You have to check the package of JDK itself, or based on the error message, do this:
locate jni.h
The output should match or compatible with your JAVAHOME, e.g:
/usr/lib/jvm/java-7-openjdk-amd64/include/jni.h
/usr/lib/jvm/java-7-oracle/include/jni.h
Update 1:
R CMD javareconf is looking for the jni.h file under $(JAVA_HOME)/include
You have JDK installed, but it is very likely that you are having default java to a JRE directory, that why the error happened.
You can see where default-java is really pointing to by doing this command:
jRealDir=$(readlink -f /usr/lib/jvm/default-java)
echo $jRealDir
# sample correct output: /usr/lib/jvm/jdk1.8.0_65
# or /usr/lib/jvm/java-8-oracle if you default to Oracle's
# now check jni.h
ls -l $jRealDir/include/jni.h
# sample expected output:
# /usr/lib/jvm/jdk1.8.0_65/include/jni.h
If the ls command failed, you have to setup so that javareconf ( and later rJava) can use the java from JDK not from JRE. You have two options:
Method 1: Do it system-wide
This is convenient, but may effect other program like the one you mentioned jmol. But don't worry, this is revertible, just re-run the command and pick the old one. Do the following command and pick the dir that has JDK:
sudo update-alternatives --config java
After that test how jmol works, if it works alright then congrat. You are now ready to test rJava. If not, try the second method below
Method 2: Do it for R only
put this in the .Rprofile under your home directory
Sys.setenv(JAVA_HOME = '/usr/lib/jvm/jdk1.8.0_65')
# this set JAVA_HOME for R to correct java home dir.
After updating or creating the .Rprofile DO restart R in Rstudio. The R CMD javareconf may still fail in this case, but it should be OK if you run it from Shell under Tools menu of Rstudio.
Regarding the installing or Rstudio from Ubuntu's stock repo. It would not make a difference for getting rJava running. Then again, I recommend installing Rstudio for its homepage because new version also has some nice features (i.e. better autocompletion, which I like the most).
Here is link on R-Bloggers that worked for me: https://www.r-bloggers.com/installing-rjava-on-ubuntu/
sudo apt-get install -y default-jre
sudo apt-get install -y default-jdk
sudo R CMD javareconf
install.packages("rJava")
I've been dealing with this exact issue, nothing in this thread or other that are similar have solved it. I'm on Ubuntu 16.04, here's how I got it to work:
apt-get install openjdk-9-jdk
rm -rf /usr/lib/jvm/default-java
ln -s /usr/lib/jvm/java-9-openjdk-amd64/ /usr/lib/jvm/default-java
You can see where the JAVA_HOME is in the error message.
Then use locate jni.h to find where is jni.h, next use soft link to link this location to $(JAVA_HOME)/include, just like #biocyberman mentioned.
This is what I did:
ln -s /usr/lib/jvm/java-8-openjdk-amd64/include/jni.h /opt/conda/include/jni.h
ln -s /usr/lib/jvm/java-8-openjdk-amd64/include/linux/jni_md.h /opt/conda/include/jni_md.h
ln -s /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so /usr/lib/
Since my JAVA_HOME is /opt/conda and I also don't have jni_md.h and -ljvm.
I am use Ubuntu 16.04.
I have a fresh install of ruby 2.1.1, a clean set of gem and I reinstalled V8 with homebrew but I get this 'unrecognized command line option' error.
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
/Users/jspooner/.rvm/rubies/ruby-2.1.2/bin/ruby extconf.rb
checking for main() in -lpthread... yes
checking for main() in -lobjc... yes
creating Makefile
make "DESTDIR=" clean
make "DESTDIR="
compiling accessor.cc
g++: error: unrecognized command line option '-rdynamic'
make: *** [accessor.o] Error 1
make failed, exit code 2
g++ version
g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/4.8.3/libexec/gcc/x86_64-apple-darwin13.2.0/4.8.3/lto-wrapper
Target: x86_64-apple-darwin13.2.0
Configured with: ../configure --build=x86_64-apple-darwin13.2.0 --prefix=/usr/local/Cellar/gcc/4.8.3 --enable-languages=c,c++,objc,obj-c++,fortran --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-cloog=/usr/local/opt/cloog --with-isl=/usr/local/opt/isl --with-system-zlib --enable-version-specific-runtime-libs --enable-libstdcxx-time=yes --enable-stage1-checking --enable-checking=release --enable-lto --disable-werror --enable-plugin --disable-nls --enable-multilib
Thread model: posix
gcc version 4.8.3 (GCC)
if Ubuntu users having this situation (e.g.: on a Ubuntu VM)
gem install therubyracer -v '0.11.4'
\Building native extensions. This could take a while...
ERROR: Error installing therubyracer:
ERROR: Failed to build gem native extension.
/home/deploy/.rbenv/versions/2.0.0-p247/bin/ruby extconf.rb
checking for main() in -lpthread... yes
creating Makefile
make "DESTDIR="
compiling accessor.cc
make: g++: Command not found
make: *** [accessor.o] Error 127
Gem files will remain installed in /home/deploy/.rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/therubyracer-0.11.4 for inspection.
Results logged to /home/deploy/.rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/therubyracer-0.11.4/ext/v8/gem_make.out
Ubuntu solution:
sudo apt-get install build-essential g++
I had to run brew update gcc.
brew upgrade gcc
==> Upgrading 1 outdated package, with result:
gcc 4.9.1
==> Upgrading gcc
==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/gcc-4.9.1.mavericks.bottle.2.tar.gz
######################################################################## 100.0%
==> Pouring gcc-4.9.1.mavericks.bottle.2.tar.gz
==> Caveats
GCC has been built with multilib support. Notably, OpenMP may not work:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60670
If you need OpenMP support you may want to
brew reinstall gcc --without-multilib
==> Summary
🍺 /usr/local/Cellar/gcc/4.9.1: 1152 files, 203M
Changing the symbolic link of g++ and gcc worked for me on Mac OS 10.10.
ls -al /opt/local/bin/gcc
ls -al /opt/local/bin/g++
sudo rm /opt/local/bin/gcc
sudo rm /opt/local/bin/g++
sudo ln -s /usr/bin/llvm-gcc /opt/local/bin/gcc
sudo ln -s /usr/bin/llvm-g++ /opt/local/bin/g++
gem install therubyracer -v '0.12.1'
sudo rm /opt/local/bin/g++
sudo rm /opt/local/bin/gcc
sudo ln -s /opt/local/bin/g++-mp-4.9 /opt/local/bin/g++
sudo ln -s /opt/local/bin/gcc-mp-4.9 /opt/local/bin/gcc
I downloaded nginx from it's site for linux(I use ubuntu 10.4).I extracted nginx-1.0.6.tar.gz and there was a configure file in that directory. So I entered "./configure" command in shell. It seemed to be configured right.After I entered "make" command ,It said this error:
make -f objs/Makefile
make[1]: Entering directory `/usr/local/nginx'
cd ./auto/lib/pcre/ \
&& if [ -f Makefile ]; then make distclean; fi \
&& CC="gcc" CFLAGS="-O2 -fomit-frame-pointer -pipe " \
./configure --disable-shared
/bin/sh: ./configure: not found
make[1]: *** [auto/lib/pcre//Makefile] Error 127
make[1]: Leaving directory `/usr/local/nginx'
make: *** [build] Error 2
what should I do now?
you have to install Dependencies .
generally these will be enough
libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev
so you can first install them
sudo apt-get install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev
and then compile .. also make sure you run the make command as root.
The ./configure program of nginx wants to find either the shared libs to build nginx dynamicaly linked or the sources of openssl prce and zlib respectivly.
The obove mentioned error occurs when you invoke ../nginx/configure with the wrong options.
--with-pcre=/path/to/lib # where libpcre.a resides
--with-openssl=/path/to/lib # where libssl.a resides
--with-zlib=/path/to/lib # where libz.a resides
is wrong especially when ld.so has no idea about these libs
If you build a statically linked version of nginx
try instead
--with-pcre=/path/to/src/of/pcre
--with-openssl=/path/to/src/of/openssl
--with-zlib=/path/to/src/of/zlib
e.g.
--with-pcre=../pcre-8.36 --with-openssl=../openssl-1.0.2 --with-zlib=../zlib-1.2.8
Download PCRE from source
Unzip it (do not install)
Copy this path to configure (from Downloads folder)
./configure --with-pcre=/home/USER/DOWNLOADS/pcre-8.37/
Enter your nginx install directory - I solved this error by editing objs/Makefile and removing -Wall and -Werror params so it looks like this (second line):
CC = gcc
CFLAGS = -pipe -O -W -Wpointer-arith -Wno-unused-parameter -Wunused-function -Wunused-variable -Wunused-value -g
Also, running your ./configure should initiate a long procedure of verifications to ensure
that your system contains all the necessary components. If the configuration fails for any reason, check
less objs/autoconf.err
for more details. Any errors at configuration are usually based on missing dependencies for your configuration.
You didn't configure it right. Use these commands (in the nginx directory):
./configure --with-pcre=./auto/lib/pcre/ --with-zlib=./auto/lib/zlib/
./configure
make
sudo make install
Look for Ubuntu installation at http://wiki.nginx.org/Install.
Look for ubuntu/ centos installation at https://nodevine.com/library/installing-multiple-virtual-hosts-on-nginx-on-ubuntu-12-04-and-cent-os-6
We can now add the repository to install the latest version of nginx:
sudo add-apt-repository ppa:nginx/stable
Note: If this command still does not work (normally on 12.10), run the following command:
sudo apt-get install software-properties-common
This will add the repository to Ubuntu and fetches the repository's key. This is to verify that the packages have not been interfered with since they have been built.
Step Three - Updating the Repositories
After adding a new repository, you will need to update the list:
sudo apt-get update
Install nginx
To install nginx or update the version you already have installed, run the following command:
sudo apt-get install nginx
Check That Nginx is Running
You can check to see that nginx is running by either going to your VPS' IP address/domain, or typing in:
service nginx status
This will tell you whether nginx is currently running.
with a vps debian wheezy
I have to install a lot of tools in order to install nginx 1.2.9 :
apt-get install libpcre3 libpcre3-dev
apt-get install --reinstall zlibc zlib1g zlib1g-dev
apt-get install make
apt-get install sudo