Activate Anaconda from a bash script? - r

I am running an R script for the first time on my university's cluster. I am using Anaconda to manage my R packages. I can successfully run the script from the command line but I get a "there is no package called _____" error when I use a bash script to call the same code.
I did a lot of searching around and found this post:
Conda command working in command prompt but not in bash script
So I changed my ~/.bashrc from:
export PATH="/home/agarbuzov/anaconda2/bin:$PATH"
to:
. /home/agarbuzov/anaconda2/etc/profile.d/conda.sh
And that didn't help. I don't have a lot of experience using conda -- I've just run a few jobs on the cluster. Any guidance would be greatly appreciated because I have no ideas here.
This is what my test script looks like:
#!/bin/csh
#PBS -q hotel
#PBS -l nodes=1:ppn=1
#PBS -l walltime=1:00:00
#PBS -N tom_bootstraps
#PBS -o tomboot_output.txt
#PBS -e tomboot_err.txt
#PBS -V
#PBS -M ***
#PBS -m abe
source /home/agarbuzov/anaconda2/etc/profile.d/conda.sh
conda activate r_env
Rscript ~/ascripts/1_rWGCNA_bootstrap_test.R
All the packages I need are listed when I call $conda list.
$conda info
active environment : r_env
active env location : /home/agarbuzov/anaconda2/envs/r_env
shell level : 1
user config file : /home/agarbuzov/.condarc
populated config files : /home/agarbuzov/.condarc
conda version : 4.6.8
conda-build version : 1.21.3
python version : 2.7.15.final.0
base environment : /home/agarbuzov/anaconda2 (writable)
channel URLs : https://conda.anaconda.org/bioconda/linux-64
https://conda.anaconda.org/bioconda/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/agarbuzov/anaconda2/pkgs
/home/agarbuzov/.conda/pkgs
envs directories : /home/agarbuzov/anaconda2/envs
/home/agarbuzov/.conda/envs
platform : linux-64
user-agent : conda/4.6.8 requests/2.21.0 CPython/2.7.15 Linux/2.6.32-696.18.7.el6.x86_64 centos/6.6 glibc/2.12
UID:GID : 520822:10494
netrc file : None
offline mode : False

I'm trying to remember where I found this, but there is a snippet of code, I don't understand what it does, but this fixes the problem.
eval "$(conda shell.bash hook)"
Alternatively, with some cluster management software (HTCondor, Slurm, ...) you can specify to run your jobs using your home environment.
Alternatively, you could also try source'ing you .bashrc in your submission script.
At any rate, the first method does seem to work.

Related

Why my zsh completion is not sourcing it?

I try to setup my zsh-completions. I also use oh-my-zsh .
A lot of shell commands provide a some_command completion zsh
and they often suggest sourcing it directly in .zshrc like so:
source <(kubeone completion zsh)
source <(tilt completion zsh)
source <(civo completion zsh)
I can't get it working. Only when I output the command into a file echo "$(kubeone completion zsh)" > ~/.dotfiles/completions/_kubeone.
(That ~/.dotfiles/completions/-path is in my fpath):
fpath=(/usr/local/share/zsh-completions $fpath)
fpath=(~/.dotfiles/completions $fpath)
But that approach is not that neat. I wanna make the use of adding
source $(my_command completion zsh) to my .zsh_rc file.
Interestingly SOME commands are working with that approach: like: source <(kompose completion zsh) but source <(kubeone completion zsh) wont???
Here is an excerpt from my ~/.zshrc:
#!/bin/sh
[ -f ~/.dotfiles/fubectl.source ] && source ~/.dotfiles/fubectl.source
export ZSH_THEME="powerlevel10k/powerlevel10k"
export DOTFILES=$HOME/.dotfiles
export ZSH=$DOTFILES/oh-my-zsh
BUNDLED_COMMANDS=(rubocop)
UNBUNDLED_COMMANDS=(berks foreman mailcatcher rails ruby spin rubocop)
plugins=(zsh-completions zsh-autosuggestions docker fzf bundler brew cask capistrano codeclimate coffee dotenv gem git github grunt helm heroku history iterm2 minikube node redis-cli redis-cli rails rake rake-fast ruby rbenv textmate macos pod zeus terraform)
source $ZSH/oh-my-zsh.sh
complete -o nospace -C /usr/local/bin/terraform terraform
# load autocomplete
fpath=(/usr/local/share/zsh-completions $fpath)
fpath=(~/.dotfiles/completions $fpath)
autoload -U compinit && compinit
source <(kompose completion zsh) # WORKS
source <(kubeone completion zsh) # DONT WORK
(kompose a→tab WORKS)
kubeone a→tab returns only my folder structure
❯ kubeone completion zsh
Application\ Support/ MAMP/ VirtualBox\ VMs/ oh-my-zsh
Applications/ Movies/ Wine\ Files/ output.json
Coding/ Music/ builds/
...
..
.
but should return:
❯ kubeone apply
apply -- Reconcile the cluster
completion -- Generates completion scripts for bash and zsh
config -- Commands for working with the KubeOneCluster configuration manifests
document -- Generates documentation
help -- Help about any command
install -- Install Kubernetes
...
..
.
What I tried so far:
after opening terminal rm ~/.zcompdump*, then tried command, then reopened terminal and tried command again.
autoload -Uz compinit && compinit -i with different arguments
I think there is some misconfiguration or maybe a brew formula is disturbing?
Here is the complete output of my brew list command, as well my complete .zshrc file: https://gist.github.com/exocode/79c9acfad9828b73d05d2abfa39f1724

How to disable "private mount namespace" (sandboxing) with the Nix package manager?

I'm trying to use nix on repl.it. I'm using static-nix from https://matthewbauer.us/blog/static-nix.html. If I run the following code:
mkdir -p "$HOME/.cache/nix/"
curl https://matthewbauer.us/nix > "$HOME/.cache/nix/nix.exe"
cat "$HOME/.cache/nix/nix.exe" | bash -s run --no-sandbox --store "$HOME/.cache/nix/store" -f channel:nixpkgs-unstable bash graphviz -c sh -c 'dot --help'
I get this error:
error: setting up a private mount namespace: Operation not permitted
I tried --no-sandbox, --option sandbox false and --option build-use-sandbox false, none of these have any effect on the error.
This is executed as non-root on a machine for which it is not possible for me to change kernel settings.
Here's a REPL reproducing the issue (it runs for a short while before displaying the error): https://repl.it/#suzannesoy/AgonizingWittyCoding#main.sh

Unable to export env variable from script

I'm currently struggling with running a .sh script I'm trying to trigger from Jenkins.
Within the Jenkins "execute shell" section, I'm connecting to a remote server (The Jenkins agent does not have right OS to build what I need.), using:
cp -r . /to/shared/drive/to/have/access/on/remote
ssh -t -t username#servername << EOF
cd /to/shared/drive/to/have/access/on/remote
source build.sh dev
exit
EOF
Inside build.sh, I'm exporting R_LIBS to build a package for different R versions.
...
for path in "${!rVersionPaths[#]}"; do
export R_LIBS="${path}"
Rscript -e 'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
...
Setting R_LIBS should functions here like setting lib within install.packages(...). For some reason the R_LIBS export doesn't get picked up. Also setting other env variables like http_proxy are ignored. This causes any requests outside the network to fail.
Is there any particular way of achieving this?
Maybe pass those variables with env, like
env R_LIBS="${path}" Rscript -e 'install.packages(c("someDependency", .....
Well i'm not able to comment on the question, so posting it as answer.
I had similar problem when calling remote shell script from Jenkins, the problem was somehow bash_profile variables were not loaded when called the script from Jenkins but locally it worked. Loading the bash profile in ssh connection solved it for me.
Add source to bash_profile in build.sh
. ~/.bash_profile OR source ~/.bash_profile
Or
Reload bash_profile in ssh connection
`ssh -t -t username#servername << EOF
. ~/.bash_profile
your commands here
exit
EOF
You can set that variable in the same command line like this:
R_LIBS="${path}" Rscript -e \
'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
It's possible to append more variables in this way. Note that this will set those environment variables only for the command being called after them (and its children processes as well).
You said that "R_LIBS export doesn't get picked up". Question Is the value UNSET? Or is it set to some other value & you are trying to override it?
It is possible that SSH may be invoking "/bin/sh -c". Based on the second answer to: Why does 'cd' command not work via SSH?, you can simplify the SSH command and explicitly invoke the build.sh script in Bash:
cp -r . /to/shared/drive/to/have/access/on/remote
ssh -t -t username#servername "cd /to/shared/drive/to/have/access/on/remote && bash -f build.sh dev"
This makes the SSH invocation more similar to invoking the command within a remote interactive shell. (You can avoid sourcing scripts and exporting variables.)
You don't need to export R_LIBSor env R_LIBS when it is possible to prefix any command with local environment variable overrides (agrees with Luis' answer):
...
for path in "${!rVersionPaths[#]}"; do
R_LIBS="${path}" Rscript -e 'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
...
The Rscript may be doing a lot with env vars. You can verify that you are setting the R_LIBS env var by replacing Rscript with the env command and observe the output:
...
for path in "${!rVersionPaths[#]}"; do
R_LIBS="${path}" env
...
According to this manual "Initialization at Start of an R Session", Rscript looks in several places to load "site and user files":
$R_PROFILE
$R_HOME/etc/Renviron
$R_HOME/etc/Renviron.site
$R_ENVIRON_USER
$R_PROFILE_USER
./.Rprofile
$HOME/.Rprofile
./.RData
The "Examples" section of that manual shows this:
## Not run:
## Example ~/.Renviron on Unix
R_LIBS=~/R/library
PAGER=/usr/local/bin/less
If you add the --vanilla command-line option to ignore all of these files, then you may get different results and know something in the site/init/environ files is affecting your R_LIBS! I cannot run this system myself. Hopefully we have given you some areas to investigate.
You probably don't want to source build.sh, just invoke it directly (i.e. remove the source command).
By source-ing the file your script is executed in the SSH shell (likely sh) rather than by bash, which it sounds like is what you intended.

Trial AppDynamics package fails to install

Getting below error during platform installation:
"Required libaio package is not found. ..."
However, above package is already installed:
rpm -q libaio
libaio-0.3.107-10.el6.x86_64
Here is output from the installation script:
./platform-setup-x64-linux-4.4.3.10393.sh
Unpacking JRE ...
Preparing JRE ...
Starting Installer ...
May 30, 2018 6:51:23 PM java.util.prefs.FileSystemPreferences$2 run
INFO: Created system preferences directory in java.home.
Verifying if the libaio package is installed. /opt/appdynamics/platform/installer/checkLibaio.sh
I got this too... I was running from command-line as a non-root user:
./platform-setup-x64-linux-4.4.3.10393.sh -q -varfile /appd/home/Install/response.varfile
I added the shell expand(-x) switch and log to the command(s) like so:
bash -x ./platform-setup-x64-linux-4.4.3.10393.sh -q -varfile /appd/home/Install/response.varfile > install.log 2>&1
If we tail the last bit of that log you get, this response in debug mode:
Verifying if the libaio package is installed. /opt/appdynamics/platform/installer/checkLibaio.sh
Required libaio package is not found. For instructions on installing
the missing package, refer to https://docs.appdynamics.com/display/PRO44/Enterprise+Console+Requirements
and the script checkLibaio.sh isn't left there... so you cannot figure it out easily. I also have a RedHat variant with the packages installed:
rpm -qa | grep libaio
libaio-0.3.109-13.el7.x86_64
Strangely enough I have one VM from the same image that will install the distribution just fine, and one that will not, so on the broken install (where I really want to install this). I ran another command from the expanded view of the install.log, which was a really long JVM command line. Anyways I got it to work and then made a looping script to retrieve the file (Because AppD for some reason removes the check script before you can look at it). The script is as follows:
#!/bin/sh
# Script used to check if the machine has libaio on it or not.
cat /dev/null > /opt/appdynamics/platform/installer/.libaio_status
chmod 777 /opt/appdynamics/platform/installer/.libaio_status
# Check if the dpkg or rpm command exists before running it.
command -v dpkg >/dev/null 2>&1
OUT=$?
if [ $OUT -eq 0 ];
then
if [ `dpkg -l | grep -i libaio* | wc -l` -gt 0 ];
then
echo SUCCESS >> /opt/appdynamics/platform/installer/.libaio_status
exit 0
fi
else
command -v rpm >/dev/null 2>&1
OUT=$?
if [ $OUT -eq 0 ];
then
if [ `rpm -qa | grep -i libaio* | wc -l` -gt 0 ];
then
echo SUCCESS >> /opt/appdynamics/platform/installer/.libaio_status
exit 0
fi
fi
fi
echo FAILURE >> /opt/appdynamics/platform/installer/.libaio_status
exit 1
I you run this script like me on the faulty platform what you will discover is that your version of Linux has both:
dpkg
and
rpm
installed. To work around this you should temporarily make a name change to one of these two package manager executables so it cannot be found (by your shell environment).
Most common here will be that you are running a RedHat variant where someone chose to install dpkg (For who knows what reason). If so desired remove that package and the install should be successful.

symfony2 app console: no available command

After HDD crash, I had to reimport my symfony2 app into Eclipse from my SVN server.
After syncing everything, I can't use the console anymore. I only get 2 commands : list and help.
I tried:
php bin/vendors install --reinstall
At the end, I got the following message:
[InvalidArgumentException]
There are no commands defined in the "assets" namespace.
[InvalidArgumentException]
There are no commands defined in the "cache" namespace.
My configuration is pretty simple:
- ubuntu server 11.04 (64bits)
- virtualbox OSE
How can I fix it?
Here is the result of app/console list command:
oc#ubuntu-server:/var/www/projets/Simoov2/src$ app/console list
Symfony version 2.0.0-RC4 - app/dev/debug
Usage:
[options] command [arguments]
Options:
--help -h Display this help message.
--quiet -q Do not output any message.
--verbose -v Increase verbosity of messages.
--version -V Display this program version.
--ansi Force ANSI output.
--no-ansi Disable ANSI output.
--no-interaction -n Do not ask any interactive question.
--shell -s Launch the shell.
--env -e The Environment name.
--no-debug Switches off debug mode.
Available commands:
help Displays help for a command
list Lists commands
OK. As I thought, this is not related to symfony2 but related to virtualbox mounting system.

Resources