I have a problem when running R scripts on a Unix cluster as a batch job. The issue is when trying to load libraries in the environment, R cannot find the library. I'll give you an example. I'll use a basic R script names sess.R:
print(.libPaths())
library("gtools")
print(sessionInfo())
If I just run this script from the command line using the command:
$ Rscript sess.R
I get the following output:
[1] "/usr/lib64/R/library" "/usr/share/R/library"
R version 3.2.3 (2015-12-10)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS release 6.6 (Final)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets base
other attached packages:
[1] gtools_3.5.0
The library "gtools" is correctly loaded, script is working. however if I write a simple batch job (I will include in the job a couple of optional parameters including an error and output file) like:
#!/bin/bash
#SBATCH --output=sess.out
#SBATCH --error=sess.err
Rscript sess.R
The job fails after a second. The two output files I get are of course sess.out and sess.err.
Sess.out contains the library directory:
[1] "/usr/lib64/R/library" "/usr/share/R/library"
which seem to be the same as when running Rscript from the command line, so no error there. However there is no printing of the Info Session since the script has been terminated. The sess.err file contains the following error:
Error in library("gtools") : there is no package called ‘gtools’
Execution halted
There, it seems like R cannot find gtools in this situation, even if the library path is the same...
Am I missing something? Is there an error I don't see somewhere? Or is it a problems of cluster settings?
Related
I'm trying to run an R script, which can be found here with the command
plot_trace.R -m ./log.model.csv ./log.trace.csv.
The two csv files can be found here.
This produces the error:
Error in `:=`(variable, as.character(variable)) :
Check that is.data.table(DT) == TRUE. Otherwise, := and `:=`(...) are defined for use in j, once only and in particular ways. See help(":=").
Calls: [ -> [.tbl_df -> check_names_df -> :=
Last but not least my sessionInfo() output:
R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.4.4
Thanks
UPDATE
I found two problems with this script:
The author mixes functions from tidyr and variables of class data.table. These functions will likely return data frames and using data.table syntax on these will cause errors. Unfortunately, some tidyr functions may return data.table when operating on one, but this seems to vary greatly depending on the version of the tidyr package.
The unnest function from tidyr has a new interface. The script won't work as is with tidyr version 1.0.0 or later.
Below are the lines I changed to fix the script, using R version 3.6.1 and tidyr version 1.0.0:
line 56: added trace.df <- as.data.table(trace.df)
line 64: changed to trace.df <- unnest(trace.df, cols)
line 65: added trace.df <- as.data.table(trace.df)
Original answer
I was able to run the script on the data you provided without error. Both
Rscript --vanilla plot_trace.R -m ./log.model.csv ./log.trace.csv
and
chmod +x plot_trace.R
./plot_trace.R -m ./log.model.csv ./log.trace.csv
produced the following plot:
Can you run the script step by step in RStudio to check the class of trace.df after each step?
It looks like the unnest call is creating something that is not a data table.
At line 56 try adding:
trace.df <- as.data.table(trace.df)
I installed Microsoft-R-Open 3.4.0 on a Red Hat Linux Enterprise 7.3 machine following the instructions https://mran.microsoft.com/documents/rro/installation/ .
R starts up and seems to be doing fine at first sight. However, when I try to list the files in a directory with the command
files <- list.files(path="/home/username/directory_name/", pattern="*.Rda",, full.names=T, recursive=FALSE)
I get the error
translateCharUTF8' must be called on a CHARSXP
Execution halted
On my local windows machine the command works fine.
Googling this nothing really comes up except that the installation might be broken.
The strange thing is, that if I copy and paste the command into R and execute it, it does not work, but if I copy and paste it into R and change that command in a way that should not change its result but only add spaces or something like that it might run. E.g. changing it to
files <- list.files(path = "/home/username/directory_name/", pattern = "*.Rda",, full.names = T, recursive = FALSE)
might work, might not work and return the same error, or might execute but when I then type "files" that might return
[1]Error: 'getCharCE' must be called on a CHARSXP
When using R from the R-foundation (https://www.r-project.org/, installed via EPEL), I get the same error and behaviour.
The command sessionInfo() returns the following:
sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.3 (Maipo)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
uname -mrs gives me:
Linux 3.10.0-514.el7.x86_64 x86_64
Any help would be greatly appreciated,
best regards
Stefan
Your files have names containing accented characters. Changing them to pure-ASCII names should fix the problem.
For Example
unicode_and_raw_filename = paste0("/tmp/\u1234", as.raw("A"))
training_rows <<- read.csv(unicode_and_raw_filename, header = FALSE)
produces:
'translateCharUTF8' must be called on a CHARSXP, but got 'raw'
Looks like R internals has some spaghetti code in charset conversion for Latin2, ISO_8859-2, UTF-8, and CP1252: https://stat.ethz.ch/R-manual/R-devel/library/base/html/iconv.html
I have a headless R server which serves the recent rstudio-server.
I need to pass it some objects via svSocket.
Because the svSocket uses TCL/TK message queue, which needs a working X11 session I know I need to embed things in a virtual X11 environment.
I log into the server via ssh and in the command line I put the following commands:
sudo Xvfb :0 &
export DISPLAY=":0"
/usr/lib/rstudio-server/bin/rserver &
R
Then I log into the rstudio-server via its web interface and type these commands:
> library(svSocket)
> startSocketServer()
[1] TRUE
then, on this R session I put the following
> library(svSocket)
> con<-socketConnection(host='localhost', port=8888)
> evalServer(con,'2+2')
I expect to get "4" result, but instead the R hangs, never returning any prompt.
If I replace the rstudio-server with the regular R, everything works correctly.
What is so special in the way it treats rsession that this example breaks? How to fix it?
SessionInfo():
R version 3.2.3 (2015-12-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=pl_PL.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=pl_PL.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=pl_PL.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=pl_PL.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
You need to mention the port number when you start the socket connection
> library(svSocket)
> startSocketServer(port=8888)
It should work after this.
I have a script that loads data, processes it, and saves the processed data out. I run it manually without a problem, but it does not work with Rscript. It uses the data.table package, and Rscript cannot find it. I have tried the solution here but get the following error message:
Error in eval(expr, envir, enclos) : could not find function ".getNamespace"
In addition: Warning message: package ‘data.table’ was built under R version 3.1.1 Error : unable to load R code in package ‘chron’
Error: package/namespace load failed for ‘data.table’
Execution halted
The script looks like:
#!/usr/local/bin/Rscript
args = commandArgs(trailingOnly = TRUE) # Take arguments from command line
library.path <- '/home/zack/R/x86_64-redhat-linux-gnu-library/3.1'
library('data.table', lib.loc = library.path)
That code is after failing with library(data.table). But I can load data.table fine, and so the rest of the script works, if I do this manually. The command I submit to bash is:
nohup /usr/local/bin/Rscript --vanilla <path to script> <args> &
Session info:
> sessionInfo()
R version 3.1.1 (2014-07-10)
Platform: x86_64-redhat-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
which Rscript:
/usr/local/bin/Rscript
/usr/local/bin/R loads data.table fine when it is called in an interactive session.
All of a sudden my system does no want to create plots from inside an R environment.
When I do just something simple like:
>$R
>X11()
(I can use whatever plotting function of R I want to. Not only X11())
A new windows pops open just like expected but then freezes. It not only freezes the window itself it also freezes the underlying R session. The only way to get out is to xkill the plot window.
After doing that inside R this message pops up:
Error in X11() : X11 fatal IO error: please save work and shut down R
I have Googled through several things but I am not able to solve this. I am working on a CentOS6.7 OS.
sessionInfo()
>R version 3.2.2 (2015-08-14)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS release 6.7 (Final)
>locale:
> [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C
> [3] LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8
> [5] LC_MONETARY=en_US.utf8 LC_MESSAGES=en_US.utf8
> [7] LC_PAPER=en_US.utf8 LC_NAME=C
> [9] LC_ADDRESS=C LC_TELEPHONE=C
>[11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C
>attached base packages:
>[1] stats graphics grDevices utils datasets methods base `
Try the following:
Include a line to use ggsave to save the plot to a specific file location
Save your script from RStudio and quit/kill your session
Open a terminal, navigate to the folder with your script, and run it with:
Rscript your_code.R
View the plot in the file location you specified in step 1 above