I am trying to install a spack package in a cluster, and if I use
spack install namd
Spack download and install its own MPI interface. Since it is a cluster, I want to take advantage of the native MPI interface, that is personalized by the admin to work fast in the particular computer. How can tell to spack to use the already installed MPI interface (i.e. openmpi or mpich)?.
BTW, I am very new with spack. Thanks!
OK, I already figure it out by reading this page from spack web. I need to create a config file with spack config edit packages and add something like
packages:
openmpi:
buildable: False
modules:
openmpi#3.1.3%gcc#8.2.0 arch=linux-x86_64-centos7: /opt/modules/mpi/gcc/8/openmpi/3.1.3
all:
compiler: [gcc#8.2.0]
providers:
mpi: [openmpi#3.1.3]
Actually, I need to load first /opt/modules/compiladores/gcc/8 to make the /opt/modules/mpi/gcc/8/openmpi/3.1.3 visible, so I need something like
packages:
openmpi:
buildable: False
modules:
openmpi#3.1.3%gcc#8.2.0 arch=linux-x86_64-centos7:
- /opt/modules/compiladores/gcc/8
- /opt/modules/mpi/gcc/8/openmpi/3.1.3
But this does not work since it needs multiple external modules and it is not possible to specify more than one (see here).
Also, spack doesn't use the external module, it creates an internal one by coping and parsing. It will ignore module dependencies or environment variables from the original external module that might be important. modules.yaml needs also to be properly configured to set or prepend this environment variables.
Related
After having set the path for the default working directory as well as my first (and only) project within RStudio options I wonder why RStudio keeps creating an empty folder named "R" within my "/home" directory every time it is started.
Is there any file I could delete/edit (eventually create) to stop this annoying behaviour and if so, where is it located ?
System: Linux Mint v. 19.3
Software: RStudio v. 1.3.959 / R version 3.4.4
Thanks in advance for any hints.
Yes, you can prevent the creation of the R directory — R is configurable via a set of environment variables.
However, setting these correctly isn’t trivial. The first issue is that many R packages are sensitive to the R version they’re installed with. If you upgrade R and try to load the existing package, it may break. Therefore, the R package library path should be specific to the R version.
On clusters, an additional issue is that the same library path might be read by various cluster nodes that run on different architectures; this is rare, but it happens. In such cases, compiled R packages might need to be different depending on the architecture.
Consequently, in general the R library path needs to be specific both to the R version and the system architecture.
Next, even if you configure an alternative path R will silently ignore it if it doesn’t exist. So be sure to manually create the directory that you’ve configured.
Lastly, where to put this configuration? One option would be to put it into the user environment file, the path of which can be specified with the environment variable R_ENVIRON_USER — it defaults to $HOME/.Renviron. This isn’t ideal though, because it means the user can’t temporarily override this setting when calling R: variables in this file override the calling environment.
Instead, I recommend setting this in the user profile (e.g. $HOME/.profile). However, when you use a desktop launcher to launch your RStudio, this file won’t be read, so be sure to edit your *.desktop file accordingly.1
So in sum, add the following to your $HOME/.profile:
export R_LIBS_USER=${XDG_DATA_HOME:-$HOME/.local/share}/R/%p-library/%v
And make sure this directory exists: re-source ~/.profile (launching a new shell inside the current one is not enough), and execute
mkdir -p "$(Rscript -e 'cat(Sys.getenv("R_LIBS_USER"))')"
The above is using the XDG base dir specification, which is the de-facto standard on Linux systems.2 The path is using the placeholders %p and %v. R will fill these in with the system platform and the R version (in the form major.minor), respectively.
If you want to use a custom R configuration file (“user profile”) and/or R environment file, I suggest setting their location in the same way, by configuring R_PROFILE_USER and R_ENVIRON_USER (since their default location, once again, is in the user home directory):
export R_PROFILE_USER=${XDG_CONFIG_HOME:-$HOME/.config}/R/rprofile
export R_ENVIRON_USER=${XDG_CONFIG_HOME:-$HOME/.config}/R/renviron
1 I don’t have a Linux desktop system but I believe that editing the Env entry to the following should do it:
Exec=env R_LIBS_USER=${XDG_DATA_HOME:-$HOME/.local/share}/R/%p-library/%v /path/to/rstudio
2 Other systems require different handling. On macOS, the canonical setting for the library location would be $HOME/Library/Application Support/R/library/%v. However, setting environment variables on macOS for GUI applications is frustratingly complicated.
On Windows, the canonical location is %LOCALAPPDATA%/R/library/%v. To set this variable, use [Environment]::SetEnvironmentVariable in PowerShell or, when using cmd.exe, use setx.
I am cannot figure out how to load MPI modules a user in Fedora.
I am using fish shell and I dont know if there will be any issues with loading a module compared to bash shell
How I am trying to load MPI modules is
lumx#localhost ~> sudo module load mpi
[sudo] password for lumx:
sudo: module: command not found
lumx#localhost ~>
Also is there a way to load them during start up so i wont have to worry about loading modules again.
Fedora distributes few MPI distributions: OpenMPI and MPICH at least.
If you just install the packages, your environment is not properly set so that you can directly use the software.
However, there is one package for each distribution, which loads the required module that sets up your environment properly. For the case of MPICH is mpich-autoload.
On the other side, module command is neither a binary nor a script file, so $PATH value should not affect. It does not require admin rights, so you can use it in user mode directly.
Finally, I recommend you to check which modules are available in your system by running
$ module avail
I managed to solve the problem.
I just added /usr/lib64/openmpi/bin/ to my user paths. I don't know if this is the right solution instead of loading modules.
set -U fish_user_paths /usr/lib64/openmpi/bin/ $fish_user_paths
I am building a debian package using dpkg.
The package has a dependency on libvirt which is not desired.
The rules file does not specify this dependency, but it is added by dpkg, I suppose due to some calls to libvirt-dev at build time.
However my package works fine without libvirt. As such, libvirt is a "Recommended" package but not "Required". How do I override this dependency and make sure it is not present in my final deb file ?
Hard to know without seeing your actual package, but I'd guess that you have a binary or shared library which is linked against libvirt. That would cause dh_shlibdeps to include libvirt in the ${shlibs:Depends} substvar.
If that's your problem, then the right fix depends on what's getting linked to libvirt. It should be straightforward to determine; just run ldd on each binary or shared library object in your package, and grep for "libvirt".
If the thing linked against libvirt is only incidental to the package, and isn't part of the main functionality, then using Recommends: would indeed be the right thing. To make dh_shlibdeps exclude that object from its dependency scanning, give it a -X option. Example target for debian/rules, assuming debhelper7-style packaging:
override_dh_shlibdeps:
dh_shlibdeps -Xname_of_your_object_to_exclude
If the thing(s) linked to libvirt actually are an important part of the package functionality, then the generated libvirt dependency is appropriate. If you still don't want it, you'll need to work out how to avoid linking against libvirt during your build.
I am trying to compile Network.HTTP (http://hackage.haskell.org/package/network) on win32/cygwin. However, it does fail with following message:
Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: HsNet.h
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
Unfortuntely it does not give more clues. The HsNet.h includes sys/uio.h which, actually should not be included, and should be configurered correctly.
Don't use cygwin, instead follow Johan Tibells way
Installing MSYS
Install the latest Haskell Platform. Use the default settings.
Download version 1.0.11 of MSYS. You'll need the following files:
MSYS-1.0.11.exe
msysDTK-1.0.1.exe
msysCORE-1.0.11-bin.tar.gz
The files are all hosted on haskell.org as they're quite hard to find in the official MinGW/MSYS repo.
Run MSYS-1.0.11.exe followed by msysDTK-1.0.1.exe. The former asks you if you want to run a normalization step. You can skip that.
Unpack msysCORE-1.0.11-bin.tar.gz into C:\msys\1.0. Note that you can't do that using an MSYS shell, because you can't overwrite the files in use, so make a copy of C:\msys\1.0, unpack it there, and then rename the copy back to C:\msys\1.0.
Add C:\Program Files\Haskell Platform\VERSION\mingw\bin to your PATH. This is neccesary if you ever want to build packages that use a configure script, like network, as configure scripts need access to a C compiler.
These steps are what Tibell uses to compile the Network package for win and I have used this myself successfully several times on most of the haskell platform releases.
It is possible to build network on win32/cygwin. And the above steps, though useful (by Jonke) may not be necessary.
While doing the configuration step, specify
runghc Setup.hs configure --configure-option="--build=mingw32"
So that the library is configured for mingw32, else you will get link or "undefined references" if you try to link or use network library.
This combined with #Yogesh Sajanikar's answer made it work for me (on win64/cygwin):
Make sure the gcc on your path is NOT the Mingw/Cygwin one, but the
C:\ghc\ghc-6.12.1\mingw\bin\gcc.exe
(Run
export PATH="/cygdrive/.../ghc-7.8.2/mingw/bin:$PATH"
before running cabal install network in the Cygwin shell)
Does R have a dependency management tool to facilitate project-specific dependencies? I'm looking for something akin to Java's maven, Ruby's bundler, Python's virtualenv, Node's npm, etc.
I'm aware of the "Depends" clause in the DESCRIPTION file, as well as the R_LIBS facility, but these don't seem to work in concert to provide a solution to some very common workflows.
I'd essentially like to be able to check out a project and run a single command to build and test the project. The command should install any required packages into a project-specific library without affecting the global R installation. E.g.:
my_project/.Rlibs/*
Unfortunately, Depends: within the DESCRIPTION: file is all you get for the following reasons:
R itself is reasonably cross-platform, but that means we need this to work across platforms and OSs
Encoding Depends: beyond R packages requires encoding the Depends in a portable manner across operating systems---good luck encoding even something simple such as 'a PNG graphics library' in a way that can be resolved unambiguously across systems
Windows does not have a package manager
AFAIK OS X does not have a package manager that mixes what Apple ships and what other Open Source projects provide
Even among Linux distributions, you do not get consistency: just take RStudio as an example which comes in two packages (which all provide their dependencies!) for RedHat/Fedora and Debian/Ubuntu
This is a hard problem.
The packrat package is precisely meant to achieve the following:
install any required packages into a project-specific library without affecting the global R installation
It allows installing different versions of the same packages in different project-local package libraries.
I am adding this answer even though this question is 5 years old, because this solution apparently didn't exist yet at the time the question was asked (as far as I can tell, packrat first appeared on CRAN in 2014).
Update (November 2019)
The new R package renv replaced packrat.
As a stop-gap, I've written a new rbundler package. It installs project dependencies into a project-specific subdirectory (e.g. <PROJECT>/.Rbundle), allowing the user to avoid using global libraries.
rbundler on Github
rbundler on CRAN
We've been using rbundler at Opower for a few months now and have seen a huge improvement in developer workflow, testability, and maintainability of internal packages. Combined with our internal package repository, we have been able to stabilize development of a dozen or so packages for use in production applications.
A common workflow:
Check out a project from github
cd into the project directory
Fire up R
From the R console:
library(rbundler)
bundle('.')
All dependencies will be installed into ./.Rbundle, and an .Renviron file will be created with the following contents:
R_LIBS_USER='.Rbundle'
Any R operations run from within this project directory will adhere to the project-speciic library and package dependencies. Note that, while this method uses the package DESCRIPTION to define dependencies, it needn't have an actual package structure. Thus, rbundler becomes a general tool for managing an R project, whether it be a simple script or a full-blown package.
You could use the following workflow:
1) create a script file, which contains everything you want to setup and store it in your projectd directory as e.g. projectInit.R
2) source this script from your .Rprofile (or any other file executed by R at startup) with a try statement
try(source("./projectInit.R"), silent=TRUE)
This will guarantee that even when no projectInit.R is found, R starts without error message
3) if you start R in your project directory, the projectInit.R file will be sourced if present in the directory and you are ready to go
This is from a Linux perspective, but should work in the same way under windows and Mac as well.