Until recently I have been using rpy2 to load an R package into a Jupyter notebook running a Python kernel, and use it to read some data from a sqlserver database. A few days ago, I started seeing this error:
import rpy2.robjects as ro
R[write to console]: Error: cons memory exhausted (limit reached?)
Following this error the Python kernel crashes and is restarted by Jupyter. I would like to understand this error, what is causing it and why it has appeared when the same code ran without errors previously.
This answer, on r-devel, suggests that it is an R error produced by exceeding a maximum number of objects:
[Rd] How to debug: Cons memory exhausted
Other than this, there seems to be very little information available about this error.
The error occurs on the first line in my code. This, and the fact that the code ran previously, makes me think something is being cached from a previous sessions. I can import the data in R (using RStudio), so I think this is specific to Jupyter/rpy2 rather than a general R issue. It is not specific to this particular notebook though - if I run the above import in another notebook, on a different kernel, I get the same error. So my questions are:
Is an excess of R Objects the only thing that causes this error, or might it be something else?
Is this likely a caching problem or am I on the wrong track here?
Where might these objects be cached? (and how can I safely clear them?)
I am using jupyterlab in a conda venv on Windows 10 in a remote Amazon workspace. I have no admin privileges. Here are some versions:
import session_info
session_info.show()
-----
rpy2 3.4.4
session_info 1.0.0
-----
Click to view modules imported as dependencies
-----
IPython 7.16.1
jupyter_client 6.1.6
jupyter_core 4.6.3
jupyterlab 2.1.5
notebook 6.0.3
-----
Python 3.7.7 (default, May 6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
Windows-10-10.0.14393-SP0
-----
Searching SO reveals many ways in which importing robjects can fail, but I can only find one about this specific error (with no answers given):
RPY2 3.3.5 Errors on import
Tangentially, it seems as though this question might be germane:
Is it safe to manually delete all files in pkgs folder in anaconda python?
Is it recommended to clear out $HOME\AppData\Local\conda\conda\pkgs\cache regularly? (UPDATE: There were some quite large files in this cache folder but clearing them did not solve the issue).
UPDATE: I see the same error in Spyder, running outside the venv, so this isn't specific to jupyter.
UPDATE: I can import rpy2. A simple import rpy2 produces no error.
UPDATE:
import rpy2.situation as rps
for row in rps.iter_info():
print(row)
gives:
rpy2 version:
3.4.4
Python version:
3.7.7 (default, May 6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
Looking for R's HOME:
Environment variable R_HOME: None
InstallPath in the registry: C:\Program Files\R\R-3.6.1
Environment variable R_USER: None
Environment variable R_LIBS_USER: None
R version:
In the PATH:
Loading R library from rpy2: OK
Additional directories to load R packages from:
None
C extension compilation:
Warning: Unable to get R compilation flags.
Environment variables are not set in rpy2. Could this mean that R_NSIZE is unset, and could this cause the error?
I have fixed this by adding system environment variables: R_HOME, R_USER, and Path. [windows10, R 4.1.0, rpy2 3.4.5, python 3.8]
For me, this problem was caused by a .Rprofile file (a start-up file for R) that was located at %USERPROFILE%\Documents\.Rprofile (on Windows 11) which contained the following R code:
if (interactive()) {
require(conflicted)
}
The initialization of the R interpreter (as done through rpy2) was not able to handle loading the conflicted library. To understand why I created that .Rprofile, please review this tidyverse blog post: https://www.tidyverse.org/blog/2018/06/conflicted/
Commenting out those three lines in the .Rprofile fixed the import rpy2.robjects crash for me.
Related
I have installed the tensorflow package.
The following commands work fine in R command line
library(tensorflow)
sess = tf$Session
However when I run the same 2 linesin Rstudio I get the following error
Error: Python module tensorflow was not found
Detected Python configuration:
...
numpy: [NOT FOUND]
I suspect its numpy related but I have no idea.
This is my Rstudio information
rstudio::versionInfo()
$version
[1] ‘0.98.1103’
$mode
[1] "desktop"
Do I need to use a different version of Rstudio or is there more setup required to get tensorflow to work?
A bit more details after reading
https://github.com/rstudio/tensorflow/issues/87
import('numpy')
Error in py_module_import(module, convert = convert) : ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: DLL load failed: The specified path is invalid.
**************** FINAL EDIT *********************
I got this to work. But I have no idea why or how.
During install_tensorflow() in Rstudio, you are prompted to add the following to your PATH so I did
usr\appdata\local\programs\python\python36\Scripts
I removed it from my PATH and successfully ran tensorflow in Rstudio.
However, I added it back subsequently and could not replicate the error.
In the end I still don't know what is broken in Rstudio or what fixed it
Please use the following command to reload dynamic library paths
sudo ldconfig
I am trying to compile h5py to run with the mpi4py that comes already compiled with ParaView. As soon as I try to import the h5py package I get the error message:
ValueError: mpi4py.MPI.Status has the wrong size, try recompiling. Expected 40, got 48
I am using Open MPI to build h5py with the --mpi option to make use of the parallel functionalities.
I am suspecting the issue being how I compiled Open MPI but I could not find a flag in the configure script that looks something like the error message.
Any help/link/comment is appreciated.
My end goal is to be able to use the geoR and sp packages in R on an AWS EC2 Linux instance. The first hurdle is getting an older version of R since 3.3 doesn't support TCL/TK (at least that I've found).
After unpacking R-3.2.5.tar.gz and following the instructions in the INSTALL file (along with these configure options), a search in the config.log shows the following for TCL and TK:
configure:32908: checking for tclConfig.sh
configure:32950: checking for tclConfig.sh in library (sub)directories
configure:33143: checking for tcl.h
conftest.c:228:17: fatal error: tcl.h: No such file or directory
#include <tcl.h>
config.status:1893: creating src/library/tcltk/DESCRIPTION
config.status:1893: creating src/library/tcltk/Makefile
config.status:1893: creating src/library/tcltk/src/Makefile
ac_cv_env_TCLTK_CPPFLAGS_set=
ac_cv_env_TCLTK_CPPFLAGS_value=
ac_cv_env_TCLTK_LIBS_set=
ac_cv_env_TCLTK_LIBS_value=
r_cv_header_tcl_h=no
TCLTK_CPPFLAGS=''
TCLTK_LIBS=''
TCL_CONFIG=''
use_tcltk='no'
R installs successfully, but running capabilities("tcltk") confirms that something went wrong, as it returns FALSE.
Following the hints here, I searched for tclConfig.sh and tkConfig.sh, and realized I hadn't installed the appropriate dependencies, namely tcl-devel, and tk-devel.
Running yum list available shows a package for tcl-devel.x86_64 but nothing for tk. I downloaded source files for TCL and TK from here, but couldn't find source files for any devel packages.
So now I'm at a loss on how to proceed. Has anyone been through this process that can shed some light on it?
UPDATE
After reading the comment below, I'm afraid I didn't do my due diligence in researching TCL/TK and R 3.3.
My very early efforts to troubleshoot the whole problem consisted of the following workflow- First, without even considering an older version of R, I ran install.packages("geoR") in 3.3.2. This resulted in installation of package 'geoR' had non-zero exit status, along with:
Error: package 'tcltk' does not have a namespace
ERROR: lazy loading failed for package 'geoR'
So, naturally, I ran install.packages("tcltk") which resulted in package 'tcltk' is not available (for R version 3.3.2)
As I mentioned before, this was very early on. I took the message literally and didn't think to look further into TCL/TK for 3.3, but instead started looking to get an earlier version of R, since I had been successfully running geoR and sp on another system that had R 3.2.5.
It seams that upon installing my custom R package code stops to work? Why?
In detail, I have a tiny R package goEnrichment (click to see the Github repo). The R package includes just just two functions and some binary data.
Also there are two test R-scripts that I run after installing the R package with R CMD INSTALL goEnrichment. One test script fails while the other works.
The only difference between the two scripts is that in the working version I require the necessary libraries and source the functions file manually, while in the other failing script I require my goEnrichment library directly.
Note, that both scripts are started directly from their directory goEnrichment/exec.
Start the working test with cd goEnrichment/exec && Rscript testGoEnrichment_works.R. Excerpt:
# This is the WORKING version
require(GOstats)
require(GSEABase)
require(RMySQL)
source( "../R/goEnrichment.R" )
Start the failing test script with cd goEnrichment/exec && Rscript testGoEnrichment_fail.R. Excerpt:
# This version FAILS
require(goEnrichment)
The error I get is somewhat uninformative to me:
'dimnames' applied to non-array.
The built R package fails on two platforms, that is on Debian Wheezy 64 with R 3.0.2 and on Mac OS X Yosemite with R 3.1.1.
I really have no clue why this happens. I checked for file format and unexpected characters using Vim. The cause does not seem to be the DESCRIPTION file either, because all libraries, goEnrichment depends on, are imported.
Does any one have a clue what causes this very weird error? Help will be much appreciated.
After much testing I found the source of the problem.
For some reason when requiring my package with
require(goEnrichment)
the generic function
summary.GOHyperGResult
is not loaded, as it should be. The method however is available when the packages, goEnrichment depends upon, are required manually. As to why this is, I had no time to investigate. I believe the here reported error is related to this bug.
Thanks to Tyler for your efforts, very much!
Does anyone have a link to clear instructions on how to install and configure the necessary latex packages to build R packages on a mac?
I have some scripts for building and checking R packages on a mac server. They seemed to work fine, but after upgrading to R 3.1.3, many of the packages started failing with
Error in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, :
Running 'texi2dvi' on 'networkVignette.tex' failed.
Messages:
sh: /usr/local/bin/texi2dvi: No such file or directory
Calls: <Anonymous> -> texi2pdf -> texi2dvi
Execution halted
I found a thread which seemed to suggest I need a more recent version texinfo (5.2) than what is installed by default. And apparently I've I've got the wrong version installed in the wrong location?
which texi2pdf
/sw/bin/texi2pdf
texi2pdf --version
texi2pdf (GNU Texinfo 5.1) 5234
(same version is reported when running system('texi2pdf --version') in R )
This thread gives a link to a texinfo 5.2 source collection:
http://r.789695.n4.nabble.com/R-CMD-build-looking-for-texi2dvi-in-the-wrong-place-R-devel-td4701706.html
But I'm not familiar with installing executable from a tar.gz file on a mac. The R mac help pages I found suggest installing MacTex, which I tried but that didn't seem to help.
** Update: ** additional discussion of related problems on R-SIG-mac mailing list:
https://groups.google.com/forum/#!topic/r-sig-mac/xjyuFdl5Ezk
Update:
Here is where I'm currently at:
I removed my /sw directory to uninstall fink and all of its packages (couldn't figure out how to upgrade it)
installed homebrew
brew install texinfo installs version 5.2 the package,
but generates the message This formula is keg-only, which means it was not symlinked into /usr/local and actually installs in in /usr/local/Cellar/texinfo/5.2/bin which means it is not on the path and R won't find it.
manually symlink each of the texi2pdf, texi2dvi , etc as vincent suggests (this is because R has the /usr/local/bin location as default in the tools::texi2dvi function?
edited the /etc/paths file on the system to add /usr/local/bin so that finds the brew installed 5.2 version before it finds it before the osx system supplied version 4.6 version. This may not be necessary because R has it hardcoded?
All of this gets rid of the "can't find texi* errors", and gives me a bunch of latex errors (which I don't see on unix and windows builds) so I'm still kind of stuck.
This seems very hackish, so there must be a cleaner way? But it sounds like stuff with tex and mac is very sketchy at the moment? https://tex.stackexchange.com/questions/208181/why-did-my-tex-related-gui-program-stop-working-in-mac-os-x-yosemite
This worked for me on Mavericks and on Yosemite:
ln -s /usr/bin/texi2dvi /usr/local/bin/texi2dvi
ln -s /usr/bin/texi2pdf /usr/local/bin/texi2pdf
On my Lion system both the command which texi2pdf at a Terminal/bash prompt and from a R.app GUI prompt tell me that I have that program in:
system("which texi2pdf")
#/opt/local/bin/texi2pdf
That is a location typical for MacPorts installation. I think the /usr/local/bin/ is what the binary R version "expects". I'm not really that UNIX savvy, but I think the you can modify the PATH environment variable so that R will be able to find your installation. (Whether it will be compatible I cannot say since so much detail is missing from your question.) My Tex installation is MacTex, which I got from https://www.tug.org/mactex/. I admit to having a cobbled-together system:
system("echo $PATH")
# /opt/local/bin:/opt/local/sbin:/sw/bin:/sw/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/usr/X11/bin:/usr/X11R6/bin
That gets set at the beginning of an R session because this is the first line in my .Rprofile-(invisible)file:
Sys.setenv(PATH="/opt/local/bin:/opt/local/sbin:/sw/bin:/sw/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/usr/X11/bin:/usr/X11R6/bin")
I think /sw/bin/ installations signify a fink install, which I have had very little success with. Simon Urbanek suggests not using any package installers, but then leaves the rest of us UNIX weenies very little in the way of worked examples of how to install that various external packages that underpin the many interesting and oh-so-useful R packages. So I feel your pain, but I'm not running for President.
So I suppose you could try this at your R console before again attempting one of the earlier unsuccessful installs:
Sys.setenv(PATH=paste( Sys.getenv()$PATH, # should be the character string of the $PATH
"/sw/bin/", sep=":")
)
Wish I could offer guarantees, but if it breaks the only guarantee is that you get to keep all the pieces.
I ran into a similar error message using Mavericks 10.9.5 (factory configured) and R 3.1.
It turns out that I didn't have pdfLaTex. I went to this page: http://tug.org/mactex/ and downloaded the MacTex installation package. It's big (>2GB) but after I installed it, my R package build problems went away.
Hope this might be helpful to anyone else who runs into this error message.