I'm currently struggling with running a .sh script I'm trying to trigger from Jenkins.
Within the Jenkins "execute shell" section, I'm connecting to a remote server (The Jenkins agent does not have right OS to build what I need.), using:
cp -r . /to/shared/drive/to/have/access/on/remote
ssh -t -t username#servername << EOF
cd /to/shared/drive/to/have/access/on/remote
source build.sh dev
exit
EOF
Inside build.sh, I'm exporting R_LIBS to build a package for different R versions.
...
for path in "${!rVersionPaths[#]}"; do
export R_LIBS="${path}"
Rscript -e 'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
...
Setting R_LIBS should functions here like setting lib within install.packages(...). For some reason the R_LIBS export doesn't get picked up. Also setting other env variables like http_proxy are ignored. This causes any requests outside the network to fail.
Is there any particular way of achieving this?
Maybe pass those variables with env, like
env R_LIBS="${path}" Rscript -e 'install.packages(c("someDependency", .....
Well i'm not able to comment on the question, so posting it as answer.
I had similar problem when calling remote shell script from Jenkins, the problem was somehow bash_profile variables were not loaded when called the script from Jenkins but locally it worked. Loading the bash profile in ssh connection solved it for me.
Add source to bash_profile in build.sh
. ~/.bash_profile OR source ~/.bash_profile
Or
Reload bash_profile in ssh connection
`ssh -t -t username#servername << EOF
. ~/.bash_profile
your commands here
exit
EOF
You can set that variable in the same command line like this:
R_LIBS="${path}" Rscript -e \
'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
It's possible to append more variables in this way. Note that this will set those environment variables only for the command being called after them (and its children processes as well).
You said that "R_LIBS export doesn't get picked up". Question Is the value UNSET? Or is it set to some other value & you are trying to override it?
It is possible that SSH may be invoking "/bin/sh -c". Based on the second answer to: Why does 'cd' command not work via SSH?, you can simplify the SSH command and explicitly invoke the build.sh script in Bash:
cp -r . /to/shared/drive/to/have/access/on/remote
ssh -t -t username#servername "cd /to/shared/drive/to/have/access/on/remote && bash -f build.sh dev"
This makes the SSH invocation more similar to invoking the command within a remote interactive shell. (You can avoid sourcing scripts and exporting variables.)
You don't need to export R_LIBSor env R_LIBS when it is possible to prefix any command with local environment variable overrides (agrees with Luis' answer):
...
for path in "${!rVersionPaths[#]}"; do
R_LIBS="${path}" Rscript -e 'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
...
The Rscript may be doing a lot with env vars. You can verify that you are setting the R_LIBS env var by replacing Rscript with the env command and observe the output:
...
for path in "${!rVersionPaths[#]}"; do
R_LIBS="${path}" env
...
According to this manual "Initialization at Start of an R Session", Rscript looks in several places to load "site and user files":
$R_PROFILE
$R_HOME/etc/Renviron
$R_HOME/etc/Renviron.site
$R_ENVIRON_USER
$R_PROFILE_USER
./.Rprofile
$HOME/.Rprofile
./.RData
The "Examples" section of that manual shows this:
## Not run:
## Example ~/.Renviron on Unix
R_LIBS=~/R/library
PAGER=/usr/local/bin/less
If you add the --vanilla command-line option to ignore all of these files, then you may get different results and know something in the site/init/environ files is affecting your R_LIBS! I cannot run this system myself. Hopefully we have given you some areas to investigate.
You probably don't want to source build.sh, just invoke it directly (i.e. remove the source command).
By source-ing the file your script is executed in the SSH shell (likely sh) rather than by bash, which it sounds like is what you intended.
Related
In an Amazon EC2 terminal, I type: `sudo nano crontab -e' to bring up the editor. I have the following (empty line at the end included):
#reboot echo "Running RMV scrape & R Shiny via: nano crontab -e"
#reboot nohup python /home/ec2-user/RMV/RMV_scrape.py &
#reboot nohup shiny-server &
#reboot service start httpd
#hourly cp -f /home/ec2-user/RMV/wait_times.csv /var/shiny-server/www/wait_times.csv
Here, I'm trying to run (a) my program, (b) apache, (c) R Shiny server and (d) a script that runs hourly to copy a file.
For some reason, this fails to run. pgrep chron does show chron runs upon startup. It shouldn't be a permissions issue because I ran crontab using sudo. I had one relative pathname in my .py script but I changed it to an absolute pathname.
I've consulted:
https://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work
http://www.unix.com/answers-to-frequently-asked-questions/13527-cron-crontab.html
Any ideas why this may not be working?
I think your problems is with the command you used to edit the crontab sudo nano crontab -e does not edit the crontab you made a file named crontab in whatever directory you were working in, but crontab files are in /var and are not intended to be edited directly. For any given user crontab -e will edit the crontab using the editor specified in the environment variable EDITOR. So to edit root's crontab the command is sudo crontab -e.
That said adding entries to root's crontab is probably not what you want. You probably want to use the system crontab for some thing like this. In almost all cases the system crontab is /etc/crontab which can be edited using sudo nano /etc/crontab. Note that for the system crontab you need to add the user of the command between the time and command sections. e.g.
#reboot root echo "Running RMV scrape & R Shiny via: nano crontab -e"
Also note that crontab uses a very minimal PATH environment variable for security reasons. If a command you issue is not on the path it will not execute. Remember to either add the paths you need to the crontab PATH (specified in the particular crontab file) or use the full path to a given executable from the (filesystem) root directory.
I am trying to modify the PATH variable of my SSH server such at a non-interactive shell command ssh myserver.com 'echo $PATH' returns the desired path. I tried modifying ~/.bashrc and ~/.profile files but they only modify PATH for when I log in to the server interactively, i.e. ssh myserver.com.
Can I change this behavior in RHEL5?
~/.bash_profile is where you want to place these variables. I don't believe that bash uses ~/.profile ( I believe it is used by csh and tcsh )
I'm quite new to cron and crontab.
I've edited the crontab file and I need to execute manually one of commands so I can try it and test it beforehand. How do I do that? If it fails, is there a mode that shows the errors?
Write a shell script that you can test.
Execute that shell script from the crontab.
Remember that cron provides barely any environment - so your script may have to fix that. In particular, your profile will not be used.
Do not get fancy with what you put in the crontab.
Build a debug mode into your shell script.
No, there isn't specifically a mode that shows errors. Usually, if the cron job witters, the output is emailed to you. That is, it sends standard output and standard error information to you if the executed command writes anything to either standard output or standard error.
On MacOS X (10.6.7), the environment I got was (via a crontab entry like 12 37 17 5 * env >/tmp/cron.env):
SHELL=/bin/sh
USER=jleffler
PATH=/usr/bin:/bin
PWD=/Users/jleffler
SHLVL=1
HOME=/Users/jleffler
LOGNAME=jleffler
_=/usr/bin/env
Of those, PWD, _ and SHLVL are handled by the shell. So, to test your script reliably in a cron-like environment, use:
(cd $HOME
env -i \
SHELL=/bin/sh \
USER=$USER \
PATH=/usr/bin:/bin \
HOME=$HOME \
LOGNAME=$LOGNAME \
/path/to/script/you/execute ...
)
The -i option to env means 'ignore all inherited enviroment'; the script will see exactly the five values specified plus anything the shell specifies automatically. With no arguments, env reports on the environment; with arguments, it adjusts the environment and executes a command.
To execute a script "manually" you first have to make it executable by doing:
$ chmod +x yourScriptName
Then do either
$ ./yourScriptName
if you execute it from its path or
$ /full/path/to/yourScriptName
from anywhere.
I have a script that looks like this
#!/bin/bash
function something() {
echo "hello world!!"
}
something | tee logfile
I have set the execute permission on this file and when I try running the file like this
$./script.sh
it runs perfectly fine, but when I run it on the command line like this
$sh script.sh
It throws up an error. Why does this happen and what are the ways in which I can fix this.
Running it as ./script.sh will make the kernel read the first line (the shebang), and then invoke bash to interpret the script. Running it as sh script.sh uses whatever shell your system defaults sh to (on Ubuntu this is Dash, which is sh-compatible, but doesn't support some of the extra features of Bash).
You can fix it by invoking it as bash script.sh, or if it's your machine you can change /bin/sh to be bash and not whatever it is currently (usually just by symlinking it - rm /bin/sh && ln -s /bin/bash /bin/sh). Or you can just use ./script.sh instead if that's already working ;)
If your shell is indeed dash and you want to modify the script to be compatible, https://wiki.ubuntu.com/DashAsBinSh has a helpful guide to the differences. In your sample it looks like you'd just have to remove the function keyword.
if your script is at your present working directory and you issue ./script.sh, the kernel will read the shebang (first line) and execute the shell interpreter that is defined. you can also call your script.sh by specifying the path of the interpreter eg
/bin/bash myscript.sh
/bin/sh myscript.sh
/bin/ksh myscript.sh etc
By the way, you can also put your shebang like this (if you don't want to specify full path)
#!/usr/bin/env sh
sh script.sh forces the script to be executed within the sh - shell.
while simply starting it from command line uses the shell-environemnt you're in.
Please post the error message for further answers.
Random though on what the error may be:
path specified in first line /bin/bash is wrong -- maybe bash is not installed?
I can't seem to set a new $PATH such that it is used when executing commands via ssh user#host command. I have tried adding export PATH=$PATH:$HOME/new_path to ~/.bashrc and ~/.profile on the remote machine, but executing ssh user#host "echo \$PATH" shows that the change has not been picked up (it shows /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games). The remote machine is running Ubuntu 8.04.
I'm sure I could hack it into /etc/profile, but that's not a clean solution and it only works when one has root access.
As grawity said, ~/.bashrc is what you want, since it is sourced by non-interactive non-login shells.
I expect the problem you're having has to do with the default Ubuntu ~/.bashrc file. It usually starts with something like this:
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
You want to put anything for non-interactive shells before this line.
Do you have an ~/.bash_login or ~/.bash_profile?
Bash in interactive mode checks for these files, and uses the first existing one, in this order:
~/.bash_profile
~/.bash_login
~/.profile
So if you have an ~/.bash_profile, then whatever changes you do to ~/.profile will be left unseen.
Bash in non-interactive mode sometimes reads the file ~/.bashrc (which is also often source'd from the interactive scripts.) By "sometimes" I mean that it is distribution-dependent: quite oddly, there is a compile-time option for enabling this. Debian enables the ~/.bashrc reading, while e.g. Arch does not.
ssh seems to be using the non-interactive mode, so ~/.bashrc should be enough. When having problems like this, I usually add a few echo's to see what files are being run.
ssh documentation says:
If command is specified, it is executed on the remote host instead of a login shell.
which is why adding to the bashrc files doesn't work. you do however have the following options:
If the PermitUserEnvironment option is set in the sshd config, you can add your PATH setting to ~/.ssh/environment
ssh remotemachine 'bash -l -c "somecommand"'
You can always say:
ssh remotemachine 'export PATH=wedontneedastinkingpath; echo $PATH'
In addition to #signpolyma answer, you will have to add your export before these lines
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
Just had the same problem myself, solved it with:
ssh user#remotehost PATH=\$HOME/bin:\$PATH\; remote-command