Firstly, I'm a newbie R, AWS and python guy. So I'm trying to get a python script with embedded R code running in AWS Lambda using rpy2. I created a Lambda package on an EC2 instance following the instructions here (modified for using python 3.4). It seems that there is something funky happening with loading the R libs using ctypes, as per the following error received in the console:
OSError: lib/libRrefblas.so: undefined symbol: xerbla_
The test file (py_test.py) looks like this:
import os
import ctypes
for file in os.listdir('lib'):
if os.path.isfile(os.path.join('lib', file)):
ctypes.cdll.LoadLibrary(os.path.join('lib', file))
os.environ["R_HOME"] = os.getcwd()
os.environ["R_USER"] = os.path.join(os.getcwd(), 'rpy2')
os.environ["R_LIBS"] = os.path.join(os.getcwd(), 'library')
os.environ["LD_LIBRARY_PATH"] = os.path.join(os.getcwd(), 'lib')
import sys
sys.path.append(os.path.join(os.getcwd(),'rpy2'))
import rpy2
from rpy2 import robjects
def test_handler(event, context):
robjects.r('''
f <- function(r, verbose=FALSE) {
if (verbose) {
cat("I am calling f().\n")
}
2 * pi * r
}
print(f(3))
''')
test_handler(None,None)
I have lib/libRrefblas.so in my virtual environment. I have scoured google looking for answers but have come up empty. Any suggestions would be greatly appreciated!
If you can get the traceback, that could help, but I suspect the problem is that it's looking for xerbla_ in the wrong place. Is xerbla_ defined in the path to RLIBS? Maybe in libR.so?
Turns out the BLAS that ships with R is corrupt. Your best bet is to make sure that BLAS and Lapack are installed on the machine you are building R on and see if you can get it to build with those libraries instead.
So steps would be to uninstall R, then run
yum -y install lapack-devel.x86_64 lapack.x86_64
yum -y install blas -devel
yum -y install R.x86_64
Check to see if it has still installed with libRrefblas.so. If it has - try deleting that file and see if it will default to the system BLAS. If you get a error because it is still looking for libRrefblas.so
rm lib/libRrefblas.so
cp /usr/lib64/libblas.so.3 lib/
mv lib/libblas.so.3 lib/libRrefblas.so
Related
I am installing mpi4py using pip using the command pip3 install mpi4py, on a centos7 system.
I get this error:
/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/bin/mpicc _configtest.o -L/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib -Lbuild/temp.linux-x86_64-3.7 -Wl,--enable-new-dtags,-R/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib -o _configtest
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: /opt/pbs/lib/libpbs.so.0: undefined reference to `sec_set_context'
failure.
removing: _configtest.c _configtest.o
error: Cannot link MPI programs. Check your configuration!!!
----------------------------------------
Does anybody have any tips?
In this case on Cheyenne's HPC cluster, you will likely need to add these arguments to your compilation/link commands -L/opt/pbs/lib -Wl,-rpath,/opt/pbs/lib -lsec or reload the ncarcompilers module.
Nevertheless, I recommend you reach out to CISL's helpdesk at NCAR for any similar problems you encounter on our system as there are dedicated teams of people to help users resolve issues like this.
I tried to compute a small example with the library SimpleHypergraphs. I followed the install instructions however, I have this error :
ERROR: "HyperNetX is not installed in Python used by this Julia. Install HyperNetX and reload SimpleHypergraphs.jl"
Here's my code:
using PyCall
using Conda
Conda.runconda(`install matplotlib --yes`)
Conda.runconda(`install networkx --yes`)
run(`$(PyCall.python) -m pip install hypernetx`)
using SimpleHypergraphs
h = Hypergraph{Float64}(5,4)
h[1:3,1] .= 1.5
h[3,4] = 2.5
h[2,3] = 3.5
h[4,3:4] .= 4.5
h[5,4] = 5.5
h[5,2] = 6.5
draw(h, HyperNetX; width=5, height=5)
I don't understand why this error occurs since hypernetx is well installed.
It seems that pandas has been added to the dependencies of hypernetx and now it will not load unless pandas is available.
Restart your Julia session and run
using Conda
Conda.runconda(`install pandas --yes`)
Now it will work as expected:
draw(h, HyperNetX; width=5, height=5)
Thanks for posting that out! - I will add it to the README of SimpleHypergraphs.
Finally, note that SimpleHypergraphs uses the Python only for the last step - plotting. All processing/computations always take part in Julia so do not worry about all those Python libraries.
I keep receiving this error when trying to derive using sympy.
I have updated sympy using Anaconda Prompt and "conda update sympy" but it made no change when trying to use the derive_by_array function in jupyter-notebook. Perhaps the update isn't registering in jupyter?
What can I do to fix this issue?
Here is a general example of the code where I receive the error:
import sympy as sp
x= sp.symbols('x')
f = x**2
sp.derive_by_array(f,x)
After removing using Anaconda Prompt then installing a newer version of Anaconda, sympy started working.
I am trying to install mlflow in R and im getting this error message saying
mlflow::install_mlflow()
Error in mlflow_conda_bin() :
Unable to find conda binary. Is Anaconda installed?
If you are not using conda, you can set the environment variable MLFLOW_PYTHON_BIN to the path of yourpython executable.
I have tried the following
export MLFLOW_PYTHON_BIN="/usr/bin/python"
source ~/.bashrc
echo $MLFLOW_PYTHON_BIN -> this prints the /usr/bin/python.
or in R,
sys.setenv(MLFLOW_PYTHON_BIN="/usr/bin/python")
sys.getenv() -> prints MLFLOW_PYTHON_BIN is set to /usr/bin/python.
however, it still does not work
I do not want to use conda environment.
how to I get past this error?
The install_mlflow command only works with conda right now, sorry about the confusing message. You can either:
install conda - this is the recommended way of installing and using mlflow
or
install mlflow python package yourself via pip
To install mlflow yourself, pip install correct (matching the the R package) python version of mlflow and set the MLFLOW_PYTHON_BIN environment variable as well as MLFLOW_BIN evn variable: e.g.
library(mlflow)
system(paste("pip install -U mlflow==", mlflow:::mlflow_version(), sep=""))
Sys.setenv(MLFLOW_BIN=system("which mlflow"))
Sys.setenv(MLFLOW_PYTHON_BIN=system("which python"))
Just ran across this, and the accepted answer by #Tomas was very helpful. I added a comment above but, for some additional context, I wanted to create a more thorough response if any other Enterprise Databricks R users run across this post trying to use the MLflow package for R on Databricks.
The Databricks MLflow quickstart guide will tell you that you need to run the following:
library(mlflow)
install_mlflow()
However, for Enterprise Databricks users, the install_mlflow() function will fail if your cluster doesn't have outside connectivity privileges (as most probably don't) and can't connect to the Anaconda repo to download the necessary packages. You'll likely get an error like this:
CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/conda-forge/linux-64/current_repodata.js
The good news is that MLflow should already be installed on your Databricks runtime. So you can reference that install instead, and then as #Tomas mentioned, use it to set your R environment variables for MLFLOW_BIN and MLFLOW_PYTHON_BIN. From there, the R MLflow API works as specified (in my experience, but ymmv).
The only catch from the above solution is that when you use the system()function in R, you need to set intern=TRUE in order capture the output of the command. The default behavior of the system() function is intern=FALSE. Thus if you do not explicitly set intern=TRUE, then the exit code 0 will be returned from your system() call (or perhaps another exit code upon an error) and Sys.setenv() will set the environment variable to 0!
### intern=True missing ###
Sys.setenv(MLFLOW_BIN=system("which mlflow"))
Sys.setenv(MLFLOW_PYTHON_BIN=system("which python"))
Example output (you can see the the environment variables did not get set correctly):
s <- Sys.getenv()
s[grep("MLFLOW", names(s))]
MLFLOW_BIN 0
MLFLOW_CONDA_HOME /databricks/conda
MLFLOW_PYTHON_BIN 0
MLFLOW_PYTHON_EXECUTABLE
/databricks/python/bin/python
MLFLOW_TRACKING_URI databricks
However, when intern=TRUE, you'll get the correct environment variables:
### intern=True set ###
Sys.setenv(MLFLOW_BIN=system("which mlflow", intern=TRUE))
Sys.setenv(MLFLOW_PYTHON_BIN=system("which python", intern=TRUE))
Example output:
s <- Sys.getenv()
s[grep("MLFLOW", names(s))]
MLFLOW_BIN /databricks/python3/bin/mlflow
MLFLOW_CONDA_HOME /databricks/conda
MLFLOW_PYTHON_BIN /databricks/python3/bin/python
MLFLOW_PYTHON_EXECUTABLE
/databricks/python/bin/python
MLFLOW_TRACKING_URI databricks
Note: This was using Databricks runtime 9.1 LTS ML. This may or may not work on other Databricks runtime configurations.
I am unable to setup & run a simple darkflow program. Infact can't even configure darkflow library:
from darkflow.net.build import TFNet
==> ModuleNotFoundError: No module named 'darkflow'
My Target is to run the following program:
from darkflow.net.build import TFNet
import cv2
options = {"model": "cfg/yolo.cfg", "load": "bin/yolo.weights", "threshold": 0.1}
tfnet = TFNet(options)
imgcv = cv2.imread("./test/dog.jpg")
result = tfnet.return_predict(imgcv)
print(result
Please suggest steps so that I could configure darkflow on Jupyter Notebook (with no GPU) and run the above code
Fixed by creating the file in ipynb file in darkflow directory (downloaded from github) and executing the following from the notebook:
!python3 setup.py build_ext --inplace
!pip install -e .
!pip install .