I'm trying to run my program using torque scheduler using mpi run. Though in my pbs file I load all the library by
export LD_LIBRARY_PATH=/path/to/library
yet it gives error i.e.
error while loading shared libraries: libarmadillo.so.3:
cannot open shared object file: No such file or directory.
I guess error lies in variable LD_LIBRARY_PATH not set in all the nodes. How would I make it work?
LD_LIBRARY_PATH is not exported automatically to MPI processes, spawned by mpirun. You should use
mpirun -x LD_LIBRARY_PATH ...
to push the value of LD_LIBRARY_PATH. Also make sure that the specified path exists on all nodes in the cluster and that libarmadillo.so.3 is available everywhere.
On some systems, your environment isn't always propagated via mpirun. You should set all those variables in your .bashrc file.
Related
I am trying to run R scripts via BAT files on Windows Command Prompt.
The scripts require a few R packages such as data.table, tidyR, etc.
For operational reasons, all required R packages and dependencies (including data.table) are installed at C:\Users\username\Documents\R\R-3.5.1\library. I am not allowed to install RStudio in this environment.
When I try
"C:\Program Files\R\R-3.5.1\bin\x64\Rscript.exe" script.R, I get an error similar to
Error in library(data.table) : there is no package called 'data.table'
Execution halted
How can I set the .libPaths via Command Prompt to point to the correct location of the packages (i.e. to C:\Users\username\Documents\R\R-3.5.1\library)?
Thank you in advance.
Disclaimer: I'm unfamiliar with R.
From R: Search paths :
The library search path is initialized at startup from the environment
variable R_LIBS (which should be a colon-separated list of directories
at which R library trees are rooted) followed by those in environment
variable R_LIBS_USER. Only directories which exist at the time will be
included.
By default R_LIBS is unset, and R_LIBS_USER is set to directory
‘R/R.version$platform-library/x.y’ of the home directory (or
‘Library/R/x.y/library’ for CRAN macOS builds), for R x.y.z.
An environment variable can be created with set VARIABLE_NAME=YOUR_VALUE batch command.
So your batch file should probably be something like this:
cd /d "C:\INSERT_PATH_TO_DIRECTORY_CONTAINING_script.R"
set "R_LIBS=C:\Users\username\Documents\R\R-3.5.1\library"
"C:\Program Files\R\R-3.5.1\bin\x64\Rscript.exe" script.R
However for portability reasons (let's say a collegue asks for a copy of your script or your computer dies) I suggest putting the script, R library and batch file in a single directory, let's say C:\Users\username\Documents\R. The batch file C:\Users\username\Documents\R\script.bat becomes:
cd /d "%~dp0"
set "R_LIBS=%~dp0R-3.5.1\library"
"%PROGRAMFILES%\R\R-3.5.1\bin\x64\Rscript.exe" "%~dpn0.R"
%PROGRAMFILES% environment variable expands to full path of program files folder, %~dp0 parameter expands to full path of a directory that holds your batch file, and %~dpn0 is a batch-file full path without extension.
Notice that %~dp0R-3.5.1 is not a typo because %~dp0 includes trailing backslash.
This way you can copy C:\Users\username\Documents\R to D:\Users\SOMEOTHERNAME\Documents\R and the script will still run.
If you create another version of your script, just copy the batch file so that it has same filename as your script but .bat extension instead of .R and it should call the new script - this has proven to be very handy when debugging and distributing scripts.
Alternatively, if you would rather install libraries separately you may want to use %HOMEDRIVE%%HOMEPATH% which expands to C:\Users\username.
Extracting proper Documents folder path, as well as R installation path is possible but requires reading the registry and thus is a bit more complicated.
Running MacOS Sierra (10.12.2), I have a program that can be executed successfully from terminal:
~$ /Users/pf/apps/cfm/bin/cfm-id
Usage: cfm-id.exe <spectrum_file> <id> <candidate_file> <num_highest> <ppm_mass_tol> <abs_mass_tol> <prob_thresh_for_prune> <param_filename> <config_filename> <score_type> <apply_postprocessing> <output_filename> <output_msp_or_mgf>
However, I need to run this program within R, using the system() command. Doing so, I get the following error:
> system("/Users/pf/apps/cfm/bin/cfm-id")
dyld: Library not loaded: libFileParsers.1.dylib
Referenced from: /Users/pf31/apps/cfm/bin/cfm-id
Reason: image not found
>
So the program apparently can't find some of the necessary libraries when it is invoked as a system command from within R.
I've set my environment variables (e.g. DYLIB_LIBRARY_PATH, etc.) in an Renviron.site file and verified these using Sys.getenv().
One hint: if I turn off the "rootless" SIP using csrutil disable, this program (and another that exhibits the same behavior) works fine both in Terminal and using system("/Users/pf/apps/cfm/bin/cfm-id") within R. Unfortunately, if I turn off SIP, this breaks yet another necessary program, such that it will not run properly either in Terminal or within R.
Another hint: if I run the same program from Terminal using sudo, I get the same behavior as with the system() call within R:
$ sudo /Users/pf/apps/cfm/bin/cfm-id
dyld: Library not loaded: libFileParsers.1.dylib
Referenced from: /Users/pf31/apps/cfm/bin/cfm-id
Reason: image not found
Abort trap: 6
Seems odd that the program would fail running as superuser. I'm guessing there is an issue with permissions somewhere, but I don't know how to run it down.
I am cannot figure out how to load MPI modules a user in Fedora.
I am using fish shell and I dont know if there will be any issues with loading a module compared to bash shell
How I am trying to load MPI modules is
lumx#localhost ~> sudo module load mpi
[sudo] password for lumx:
sudo: module: command not found
lumx#localhost ~>
Also is there a way to load them during start up so i wont have to worry about loading modules again.
Fedora distributes few MPI distributions: OpenMPI and MPICH at least.
If you just install the packages, your environment is not properly set so that you can directly use the software.
However, there is one package for each distribution, which loads the required module that sets up your environment properly. For the case of MPICH is mpich-autoload.
On the other side, module command is neither a binary nor a script file, so $PATH value should not affect. It does not require admin rights, so you can use it in user mode directly.
Finally, I recommend you to check which modules are available in your system by running
$ module avail
I managed to solve the problem.
I just added /usr/lib64/openmpi/bin/ to my user paths. I don't know if this is the right solution instead of loading modules.
set -U fish_user_paths /usr/lib64/openmpi/bin/ $fish_user_paths
I work in an environment where linking of dynamic libraries are restricted to certain locations. When I use RStudio and request a new C++ file I get the "Hello World" template. When I try to compile that and link that in by clicking on "Source" in RStudio, I get an error:
LoadLibrary failure: Access is denied.
This error is because the library was located in a space which is not allowed to be able to load DLL files. To maneuver around this limitation, I would like to determine how to tell RCpp to place the temporary dll's (not in a package) in a specific location.
I know that Dirk has suggested that this is not in the scope of RCpp and that all code should live in packages, but that will not be he most user friendly environment for the users here. I suspect that most will use RStudio projects with GIT.
So, that being said, is there an environment variable that I can mangle to get RCpp to place temporary dll files in a specific place. Or is there some other mechanism which I can use to alter this?
Try setting TMPDIR which R respects. This is indeed not an Rcpp issue but a generic R CMD build / R CMD INSTALL issue.
From help(tempfile):
The environment variables TMPDIR, TMP and TEMP are checked in
turn and the first found which points to a writable directory is
used: if none succeeds /tmp is used.
PS Rcpp with lower-case C.
The documentation states that there is a command-line shell for sqlite3:
To start the sqlite3 program, just type "sqlite3" followed by the name the file that holds the SQLite database."
When I try this, in the Windows Command Prompt I get the error message:
'sqlite3' is not recognized as an internal or external command,
operable program or batch file.
Windows explorer reveals several 'Sqlite3" folders in various places:
backends(C:/Python26/Lib/site-packages/django/db)
Lib(C:/Python26)
backends(C:/Django-1.1.1/Django-1.1.1/build/lib/django/db
backends(C:/Django-1.1.1/Django-1.1.1/django/db)
How do I access the shell, can anyone help?
Download sqlite3 binary for windows here. Unzip it and put it somewhere in your path.
That's the error message you get if you try to run any executable that's not in your current directory or in the path.
To correct the problem, find the SQLite executable (SQLITE3.EXE), and run it from the directory in which it resides, or add SQLITE3.EXE to your PATH environment variable.
You have to properly set your PATH environment variable to include one of the locations where sqlite3.exe resides. Usually SQLite seems to set that environment variable upon install but the list of paths where you found it indicates that it just came as a library for various other applications. Therefore it's not too surprising that the path isn't set.
I have sqlite3 on my machine, and as others have mentioned it must be located within a folder specified by your PATH environment variable. Since I use it a lot, I threw it in windows\system32, which is where I place a lot of utilities like pstools.