Here is my environment yml file :-
# Conda environment specification. The dependencies defined in this file will
# be automatically provisioned for runs with userManagedDependencies=False.
# Details about the Conda environment file format:
# https://conda.io/docs/user-guide/tasks/manage-environments.html#create-env-file-manually
name: project_environment
dependencies:
# The python interpreter version.
# Currently Azure ML only supports 3.5.2 and later.
- python
- pip
- pip:
# Required packages for AzureML execution, history, and data preparation.
- azureml-defaults
- scikit-learn
- numpy
- azureml-monitoring
- cython
- setuptools_rust
And it is failing on below code:
-
deployment_config = AksWebservice.deploy_configuration(auth_enabled=False, collect_model_data=True, enable_app_insights=True, cpu_cores = 2, memory_gb = 2)
aks_target = AksCompute(ws,aks_name)
(On below line getting error)
service = Model.deploy(ws, service_name, [model], inference_config, deployment_config, aks_target)
service.wait_for_deployment(show_output = True)
Similar issue happened when the setuptools is not installed or not installed in the same path. Your setup.py file needs setuptools. Uninstall and install may help, it fix my issue.
pip uninstall setuptools
and then:
pip install setuptools
Also, sometimes pip version can cause this issue as well.
pip3 install -U pip
Related
I'm working through a Python exercise using Azure Machine Learning notebooks. I'm unable to import torch even after !pip install torch.
Notebook says Requirement already satisfied, then errors out with:
!pip install torch
import torch
data = torch.tensor(encode(text), dtype=torch.long)
print(data.shape, data.dtype)
print(data[:100])
4 sec
ModuleNotFoundError: No module named 'torch'
Requirement already satisfied: torch in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (1.12.0)
Requirement already satisfied: typing-extensions in /anaconda/envs/azureml_py38/lib/python3.8/site-packages (from torch) (4.4.0)
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Input In [22], in <cell line: 4>()
1 # Encode the entire dataset and store it into a torch.Tensor
3 get_ipython().system('pip install torch')
----> 4 import torch
5 data = torch.tensor(encode(text), dtype=torch.long)
6 print(data.shape, data.dtype)
ModuleNotFoundError: No module named 'torch'
I opened up a terminal in Azure ML Studio and tried pip install torch too, same Requirement already satisfied message showed.
How do I get torch (and any other Python modules where this occurs) working in AML notebooks?
I've found creating environments and installing packages through the terminal to be a much more reliable experience than doing it from an AML notebook.
I suggest using one of the provided terminals (either the one available in the compute instance's details, or the one available in JupyterLab) to create a new conda environment which you can customize to your liking.
Something like:
conda create -n my_tutorial python=3.10
conda activate my_tutorial
pip install --user ipykernel
python -m ipykernel install --user --name=my_tutorial
# Do a complete install of PyTorch, take a look at the available versions here https://pytorch.org/get-started/previous-versions/
conda install pytorch=1.13 torchvision=0.14 torchaudio=0.13 pytorch-cuda=11.7 -c pytorch -c nvidia -y
Afterwards, just make sure your notebooks use the my_tutorial kernel and you should be good to go. Whenever you want to pip install something new, just go back to the terminal, activate your kernel, and install the thing, then it should be available in your notebooks as well.
The issue was lined out in the docs here. Don't use !pip within the notebook. Instead use %pip.
I am encountering a segfault when I make a reticulated call to
matplotlib.pyplot.plot().
Steps to produce error:
Create a Dockerfile with the contents:
FROM rocker/r-ver:latest
RUN apt update && apt install -y python3.8-venv python3.8-dev
RUN install2.r --error reticulate
COPY test.R /root/
Create a file test.R (in the same location) with the contents:
reticulate::virtualenv_create(
envname = "./venv",
packages = c("matplotlib")
)
reticulate::use_virtualenv("./venv")
reticulate::py_run_string("import matplotlib.pyplot as plt; plt.plot([1, 2, 3], [1, 2, 3])")
Build an image from the Dockerfile: docker build . --tag="segfault-reprex"
Try to run test.R in the running container: docker run segfault-reprex Rscript /root/test.R. This gives the full traceback listed below.
Full traceback
Using Python: /usr/bin/python3.8
Creating virtual environment './venv' ... Done!
Installing packages: 'pip', 'wheel', 'setuptools', 'matplotlib'
Collecting pip
Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
Collecting wheel
Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting setuptools
Downloading setuptools-60.5.0-py3-none-any.whl (958 kB)
Collecting matplotlib
Downloading matplotlib-3.5.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.3 MB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting fonttools>=4.22.0
Downloading fonttools-4.28.5-py3-none-any.whl (890 kB)
Collecting packaging>=20.0
Downloading packaging-21.3-py3-none-any.whl (40 kB)
Collecting cycler>=0.10
Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting numpy>=1.17
Downloading numpy-1.22.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Collecting pillow>=6.2.0
Downloading Pillow-9.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Collecting python-dateutil>=2.7
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pyparsing>=2.2.1
Downloading pyparsing-3.0.6-py3-none-any.whl (97 kB)
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pip, wheel, setuptools, kiwisolver, fonttools, pyparsing, packaging, cycler, numpy, pillow, six, python-dateutil, matplotlib
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Uninstalling pip-20.0.2:
Successfully uninstalled pip-20.0.2
Attempting uninstall: setuptools
Found existing installation: setuptools 44.0.0
Uninstalling setuptools-44.0.0:
Successfully uninstalled setuptools-44.0.0
Successfully installed cycler-0.11.0 fonttools-4.28.5 kiwisolver-1.3.2 matplotlib-3.5.1 numpy-1.22.0 packaging-21.3 pillow-9.0.0 pip-21.3.1 pyparsing-3.0.6 python-dateutil-2.8.2 setuptools-60.5.0 six-1.16.0 wheel-0.37.1
Virtual environment './venv' successfully created.
*** caught segfault ***
address 0x7ffaeabe1100, cause 'memory not mapped'
Traceback:
1: py_run_string_impl(code, local, convert)
2: reticulate::py_run_string("import matplotlib.pyplot as plt; plt.plot([1, 2, 3], [1, 2, 3])")
An irrecoverable exception occurred. R is aborting now ...
Things I have noted:
A minimal example inovling eg. the pandas package, rather than matplotlib,
runs successfully. ie. if test.R contains:
reticulate::virtualenv_create(
envname = "./venv",
packages = c("pandas")
)
reticulate::use_virtualenv("./venv")
reticulate::py_run_string("import pandas as pd; df = pd.DataFrame()")
If you enter the container interactively (docker run -it segfault-reprex /bin/bash),
run test.R (Rscript /root/test.R), activate the resulting
virutalenv (source /root/venv/bin/activate), you can use matplotlib fine from
python (python -c "import matplotlib.pyplot as plt; plt.plot([1, 2, 3], [1, 2, 3])")
The reticulate documentation states that:
for reticulate to bind to a version of Python it must be compiled with shared
library support (i.e. with the --enable-shared flag)
docker run -it segfault-reprex /usr/bin/python3 -c "import sysconfig; print(sysconfig.get_config_vars('Py_ENABLE_SHARED'))"
shows that the container's Python was compiled with shared library support
The problem is that the R binary in rocker/r-ver:latest is compiled against a different BLAS library to the one which the numpy on PyPI is compiled against.
This was explained to me by Tomasz Kalinowski here.
The solution is to ensure numpy uses the same BLAS libraries as rocker/r-ver's R binary does. An easy way to ensure this is to compile numpy from source. This compilation could be performed at either image build-time or container runtime.
Compiling numpy at runtime
To compile numpy at container runtime we can leave our Dockerfile as is, and add a call to system2() after our initial call to reticulate::virtualenv_create(). Altering test.R to become:
reticulate::virtualenv_create(
envname = "./venv",
packages = c("matplotlib")
)
system2("./venv/bin/pip3", c("install",
"--no-binary='numpy'",
"numpy",
"--ignore-installed"))
reticulate::use_virtualenv("./venv")
reticulate::py_run_string("import matplotlib.pyplot as plt;plt.plot([1, 2, 3], [1, 2, 3])")
After rebuilding our image, we can run test.R in this container without segfault!
Compiling numpy at build-time
Compiling numpy at runtime adds ~3 mins to every call of our R script!
A better solution could be to perform this compilation at image build-time. This would mean we'd only have to wait those ~3 minutes once (at image build time), rather than every time we run our script!
A Dockerfile to do so could look like:
FROM rocker/r-ver:latest
RUN apt update && apt install -y python3 python3-dev python3-venv
RUN install2.r --error reticulate
# Create a venv
RUN python3 -m venv /root/venv
# Compile numpy from source into venv
RUN /root/venv/bin/pip3 install --no-binary="numpy" numpy --ignore-installed
COPY test.R /root/
The accompanying test.R file would then make use of reticulate::virtualenv_install() as:
reticulate::virtualenv_install(
envname = "/root/venv",
packages = c("matplotlib")
)
reticulate::use_virtualenv("/root/venv")
reticulate::py_run_string("import matplotlib.pyplot as plt;plt.plot([1, 2, 3], [1, 2, 3])")
NB. when running a container from the image with numpy already compiled, you'll need to run as either root (-u="root"), or else change the permissions on the compiled numpy version in the Dockerfile; otherwise you will encounter a permissions error.
I am deploying multiple R versions on multiple virtual desktops. I've built 3.6.3 and 4.1.2 R from source on Ubuntu 18.04.3 LTS. None of them finds the system-wide Rprofile.site file in /etc/R or the system certificates in /usr/share/ca-certificates. However R (3.4.4) installed with APT has no such problems. I used Ansible, but for the sake of this question I reproduced the deployment for one host with a shell script.
#!/bin/bash
set -euo pipefail
# install build dependecies
(command -v apt && apt-get build-dep r-base) || (command -v dnf && dnf builddep R)
version='4.1.2'
major_version=$(echo "$version" | cut -c 1)
wget "https://cran.rstudio.com/src/base/R-$major_version/R-$version.tar.gz"
tar -xzf R-$version.tar.gz
cd R-$version
./configure \
--prefix=/opt/R/$version \
--sysconfdir=/etc/R \
--enable-R-shlib \
--with-pcre1 \
--with-blas \
--with-lapack
make -j 8
make install
Note: It should run on most Linux distros with APT or RPM package managers. Increase the -j argument of make, if you have the enough cores, but no time.
So I defined the installation prefix as /opt/R/$version , but I want it read config files from /etc/R (defined --sysconfdir=/etc/R). However when I open the R interactive shell (/opt/R/4.1.2/bin/R) to try install a package:
install.packages("remotes")
then I will be prompted to choose a R package mirror, but one already defined in /etc/R/Rprofile.site:
local({
r <- getOption("repos")
r["CRAN"] <- "https://cloud.r-project.org"
options(repos = r)
})
I can force the R shell to find the Rprofile.site file by defining it with the R_PROFILE environment variable.
export R_PROFILE=/etc/R/Rprofile.site
/opt/R/4.1.2/bin/R
then call install.packages("remotes") again in the R shell. Now no mirror selection prompt will be shown, but the following error:
Warning: unable to access index for repository https://cloud.r-project.org/src/contrib:
cannot open URL 'https://cloud.r-project.org/src/contrib/PACKAGES'
Warning message:
package ‘remotes’ is not available for this version of R
A version of this package for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages
So it cannot access the repository index (real problem), then concludes that the package ‘remotes’ package is not available for my R version. Which is BS, since it was not able to read the index at the first place. So I tried a simple HTTP call in the same R shell.
curlGetHeaders("https://example.com")
and got this error:
Error in curlGetHeaders("https://example.com") : libcurl error code 77:
unable to access SSL/TLS CA certificates
So it cannot find the CA certificates in /usr/share/ca-certificates.
Since the R installed by APT has none of these problems. The compiled R does not search the right places. Even if I omit the --sysconfdir=/etc/R build option and copy or symlink the /etc/R directory under the prefix, so it will be at /opt/R/4.1.2/etc. It will still not find its config files.
The greater problem that I do not even no know how to specify the /usr/share so it may find the certificates. The rsharedir build options (the -- also missing in the makefile) will not do, because it should point to /usr/share/R/ not /usr/share, which would be a bad practice anyway.
I also tried all of this it with the 3.6.3 R version and got the same results.
Questions: How can I make the compiled R installations to find the system-wide or any config files and the certificates.
Update 1
I ran the build script on a Ubuntu server which I do not manage with the same Ansible code. On both of them R successfully finds the certificates. So the problem is not with the build script but the system state.
Update 2
I created a simple R script (install-r-package.R) which install a package:
install.packages("renv", repos="https://cran.wu.ac.at/")
then I executed it with Rscript and traced which file do they open on both the correct and erroneous hosts:
strace -o strace.log -e trace=open,openat,close,read,write,connect,accept ./Rscript install-r-package.R
It turned out that on the problematic system R does not even try to open the certificate files.
The relevant trace snippet on the correct system:
connect(5, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("137.208.57.37")}, 16) = -1 EINPROGRESS (Operation now in progress)
openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 6
read(6, "-----BEGIN CERTIFICATE-----\nMIIH"..., 200704) = 200704
read(6, "--\n", 4096) = 3
read(6, "", 4096) = 0
close(6) = 0
on the problematic system:
connect(5, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("137.208.57.37")}, 16) = -1 EINPROGRESS (Operation now in progress)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so", O_RDONLY|O_CLOEXEC) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220A\0\0\0\0\0\0"..., 832) = 832
close(6) = 0
In both cases R connects to the mirror (137.208.57.37) after that on the correct system it reads the ca-certificates.crt certificate file and many other .crt files after that. However the erroneous system jump this step altogether.
Finally I found the solution:
Since both system has the arch and OS. I cross copied the R compiled installations between them. The R which was compiled on the problematic system, but was run on the correct one gave the warnings below after the calling of the install.packages("renv", repos="https://cran.wu.ac.at/")
Warning: unable to access index for repository https://cran.wu.ac.at/src/contrib:
internet routines cannot be loaded
Warning messages:
1: In download.file(url, destfile = f, quiet = TRUE) :
unable to load shared object '/opt/R/4.1.2/lib/R/modules//internet.so':
libcurl-nss.so.4: cannot open shared object file: No such file or directory
2: package ‘remotes’ is not available for this version of R
A version of this package for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages
If I do the reverse then the installation works.
The libcurl-nss.so.4: cannot open shared object file: No such file or directory line gave me the clue that different libcurl4 flavors was used as build dependecies. I checked which dev dependecies were installed on the systems and libcurl4-nss-dev 7.58.0-2ubuntu3 were installed on the problematic system and libcurl4-gnutls-dev 7.58.0-2ubuntu3.16 on the correct system.
So I purged libcurl4-gnutls-dev from the problematic system:
apt purge libcurl4-nss-dev -y
and installed libcurl4-gnutls-dev:
aptitude install libcurl4-gnutls-dev
I used aptitude, because I had to downgrade libcurl3-gnutls 7.58.0-2ubuntu3.16 (now) -> 7.58.0-2ubuntu3 (bionic) which is a dependency of libcurl4-gnutls-dev, then I run a make clean in the R-4.1.2 source directory. Finally I re-run the build script from the question, and got a well working R, which can read the certificates, hence can reach the HTTPS using package mirrors.
I am having trouble installing R packages in JupyterLab in AI Platform on Google Cloud.
I am the owner of the project I work in.
I have created a new R 3.6 instance with the permission set to the default Compute Engine default service account.
The issue is that I for some reason do not have write access for the folder where packages are saved even though I am project owner and therefore should have write access to everything in the project.
Here is what I have tried and the error message I get:
install.packages("RCurl", repos='http://cran.us.r-project.org')
And this is the error message I get:
Warning message in install.packages("RCurl", repos = "http://cran.us.r-project.org"):
“'lib = "/opt/conda/lib/R/library"' is not writable”
Error in install.packages("RCurl", repos = "http://cran.us.r-project.org"): unable to install packages
Traceback:
1. install.packages("RCurl", repos = "http://cran.us.r-project.org")
2. stop("unable to install packages")
I have tried with both setting the repos argument and not setting it.
By design the default jupyter user does not have the root access because you are supposed to install packages locally. For example (be aware that you can replace /tmp with a local directory):
install.packages("leaflet", lib="/tmp")
I was having the exact same problem this morning. My co-worker shared his method of using R in jupypter on google cloud. In case it helps:
Create a regular compute instance, then install conda, and run the following commands:
conda create --name r_3 r-base r-essentials r-devtools jupyterlab r-devtools
conda activate r_3
nohup jupyter lab --no-browser --port=12345 --ip='0.0.0.0' &
on google cloud shell
gcloud compute ssh --project "your-project-name" --zone "your-zone" "your-instance-name" -- -L 12345:localhost:12345 -4
then click the link to the jupyter notebook after running:
tail nohup.out
My system right now is running Python 3.5.2 with pip version 19.0.3 and I'm trying to install OpenMDAO using pip. I run the command like it says on the getting started page of the website to install:
pip install openmdao
It begins to runs but then I get the following errors:
error: can't copy 'openmdao/devtools/xdsm_viewer/XDSMjs/build': doesn't exist or not a regular file
Failed building wheel for openmdao"
Any suggestions to fix this?
I am not able to replicate your issue. Are you starting from a clean environment?
Here is my attempt:
~ $ conda create -n O35 python=3.5 numpy scipy --quiet
Solving environment: ...working... done
## Package Plan ##
environment location: /home/swryan/anaconda2/envs/O35
added / updated specs:
- numpy
- python=3.5
- scipy
The following NEW packages will be INSTALLED:
blas: 1.0-mkl
ca-certificates: 2019.1.23-0
certifi: 2018.8.24-py35_1
intel-openmp: 2019.1-144
libedit: 3.1.20181209-hc058e9b_0
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 8.2.0-hdf63c60_1
libgfortran-ng: 7.3.0-hdf63c60_0
libstdcxx-ng: 8.2.0-hdf63c60_1
mkl: 2018.0.3-1
mkl_fft: 1.0.6-py35h7dd41cf_0
mkl_random: 1.0.1-py35h4414c95_1
ncurses: 6.1-he6710b0_1
numpy: 1.15.2-py35h1d66e8a_0
numpy-base: 1.15.2-py35h81de0dd_0
openssl: 1.0.2r-h7b6447c_0
pip: 10.0.1-py35_0
python: 3.5.6-hc3d631a_0
readline: 7.0-h7b6447c_5
scipy: 1.1.0-py35hfa4b5c9_1
setuptools: 40.2.0-py35_0
sqlite: 3.27.2-h7b6447c_0
tk: 8.6.8-hbc83047_0
wheel: 0.31.1-py35_0
xz: 5.2.4-h14c3975_4
zlib: 1.2.11-h7b6447c_3
Proceed ([y]/n)?
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
~ $ conda activate O35
(O35) ~ $ pip install --upgrade pip
Collecting pip
Using cached https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl
mkl-random 1.0.1 requires cython, which is not installed.
Installing collected packages: pip
Found existing installation: pip 10.0.1
Uninstalling pip-10.0.1:
Successfully uninstalled pip-10.0.1
Successfully installed pip-19.0.3
(O35) ~ $ pip install openmdao
Collecting openmdao
Collecting pyparsing (from openmdao)
Using cached https://files.pythonhosted.org/packages/de/0a/001be530836743d8be6c2d85069f46fecf84ac6c18c7f5fb8125ee11d854/pyparsing-2.3.1-py2.py3-none-any.whl
Collecting six (from openmdao)
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting pyDOE2 (from openmdao)
Collecting networkx>=2.0 (from openmdao)
Requirement already satisfied: scipy in ./anaconda2/envs/O35/lib/python3.5/site-packages (from openmdao) (1.1.0)
Requirement already satisfied: numpy in ./anaconda2/envs/O35/lib/python3.5/site-packages (from openmdao) (1.15.2)
Collecting decorator>=4.3.0 (from networkx>=2.0->openmdao)
Using cached https://files.pythonhosted.org/packages/f1/cd/7c8240007e9716b14679bc217a1baefa4432aa30394f7e2ec40a52b1a708/decorator-4.3.2-py2.py3-none-any.whl
Installing collected packages: pyparsing, six, pyDOE2, decorator, networkx, openmdao
Successfully installed decorator-4.3.2 networkx-2.2 openmdao-2.6.0 pyDOE2-1.1.2 pyparsing-2.3.1 six-1.12.0
(O35) ~ $