I have a dataframe I would like to write to an S3 bucker. I am using aws.s3 for this task. My script looks like the following.
library(aws.s3)
# set up AWS credentials
Sys.setenv("AWS_ACCESS_KEY_ID" = "ASUPERSECRETSTRING",
"AWS_SECRET_ACCESS_KEY" = "ASUPERSECRETSTRING",
"AWS_DEFAULT_REGION" = "us-east-somwhere")
s3write_using(my_data, FUN = write.csv,
bucket = "www.My_bucket",
object = unique_name)
I don't have any problems with the above script, but I don't like hard coding my AWS credentials. What can I do to prevent this?
Pass your ENV to docker run command
sudo docker run -dit -e AWS_ACCESS_KEY_ID='your_key' -e AWS_SECRET_ACCESS_KEY='your_secret' -e AWS_DEFAULT_REGION='bucket_region' busybox sh
Then modify a bit your script.
test_env=Sys.getenv(c("R_HOME"))
AWS_ACCESS_KEY_ID=Sys.getenv("AWS_ACCESS_KEY_ID");
AWS_SECRET_ACCESS_KEY=Sys.getenv("AWS_SECRET_ACCESS_KEY")
AWS_SECRET_ACCESS_KEY=Sys.getenv("AWS_DEFAULT_REGION")
message("test env is:",test_env)
Above code will get env and also one test env. If you still need to call Sys.setenv then you can pass like this once you get env.
Sys.setenv("AWS_ACCESS_KEY_ID" = AWS_ACCESS_KEY_ID,
"AWS_SECRET_ACCESS_KEY" = AWS_SECRET_ACCESS_KEY,
"AWS_DEFAULT_REGION" = AWS_SECRET_ACCESS_KEY)
Related
I'm currently struggling with running a .sh script I'm trying to trigger from Jenkins.
Within the Jenkins "execute shell" section, I'm connecting to a remote server (The Jenkins agent does not have right OS to build what I need.), using:
cp -r . /to/shared/drive/to/have/access/on/remote
ssh -t -t username#servername << EOF
cd /to/shared/drive/to/have/access/on/remote
source build.sh dev
exit
EOF
Inside build.sh, I'm exporting R_LIBS to build a package for different R versions.
...
for path in "${!rVersionPaths[#]}"; do
export R_LIBS="${path}"
Rscript -e 'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
...
Setting R_LIBS should functions here like setting lib within install.packages(...). For some reason the R_LIBS export doesn't get picked up. Also setting other env variables like http_proxy are ignored. This causes any requests outside the network to fail.
Is there any particular way of achieving this?
Maybe pass those variables with env, like
env R_LIBS="${path}" Rscript -e 'install.packages(c("someDependency", .....
Well i'm not able to comment on the question, so posting it as answer.
I had similar problem when calling remote shell script from Jenkins, the problem was somehow bash_profile variables were not loaded when called the script from Jenkins but locally it worked. Loading the bash profile in ssh connection solved it for me.
Add source to bash_profile in build.sh
. ~/.bash_profile OR source ~/.bash_profile
Or
Reload bash_profile in ssh connection
`ssh -t -t username#servername << EOF
. ~/.bash_profile
your commands here
exit
EOF
You can set that variable in the same command line like this:
R_LIBS="${path}" Rscript -e \
'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
It's possible to append more variables in this way. Note that this will set those environment variables only for the command being called after them (and its children processes as well).
You said that "R_LIBS export doesn't get picked up". Question Is the value UNSET? Or is it set to some other value & you are trying to override it?
It is possible that SSH may be invoking "/bin/sh -c". Based on the second answer to: Why does 'cd' command not work via SSH?, you can simplify the SSH command and explicitly invoke the build.sh script in Bash:
cp -r . /to/shared/drive/to/have/access/on/remote
ssh -t -t username#servername "cd /to/shared/drive/to/have/access/on/remote && bash -f build.sh dev"
This makes the SSH invocation more similar to invoking the command within a remote interactive shell. (You can avoid sourcing scripts and exporting variables.)
You don't need to export R_LIBSor env R_LIBS when it is possible to prefix any command with local environment variable overrides (agrees with Luis' answer):
...
for path in "${!rVersionPaths[#]}"; do
R_LIBS="${path}" Rscript -e 'install.packages(c("someDependency", "someOtherDependency"), repos="http://cran.r-project.org");'
...
The Rscript may be doing a lot with env vars. You can verify that you are setting the R_LIBS env var by replacing Rscript with the env command and observe the output:
...
for path in "${!rVersionPaths[#]}"; do
R_LIBS="${path}" env
...
According to this manual "Initialization at Start of an R Session", Rscript looks in several places to load "site and user files":
$R_PROFILE
$R_HOME/etc/Renviron
$R_HOME/etc/Renviron.site
$R_ENVIRON_USER
$R_PROFILE_USER
./.Rprofile
$HOME/.Rprofile
./.RData
The "Examples" section of that manual shows this:
## Not run:
## Example ~/.Renviron on Unix
R_LIBS=~/R/library
PAGER=/usr/local/bin/less
If you add the --vanilla command-line option to ignore all of these files, then you may get different results and know something in the site/init/environ files is affecting your R_LIBS! I cannot run this system myself. Hopefully we have given you some areas to investigate.
You probably don't want to source build.sh, just invoke it directly (i.e. remove the source command).
By source-ing the file your script is executed in the SSH shell (likely sh) rather than by bash, which it sounds like is what you intended.
I created a session and can execute the ls command
library(ssh)
session<-ssh_connect("someuser#x.x.x.x", keyfile = "my private key")
ssh_exec_wait(session, command = 'ls /home/')
This results of the ls command displayed in rstudio console and the function returns 0.
So this does not work
varFileList<-ssh_exec_wait(session, command = 'ls /home/')
How do I accomplish this?
Found the answer
varFileList<-capture.output(ssh_exec_wait(session, command = 'ls /home/'))
I am defining a shell script in one of the stages in my Jenkinsfile. How can I access a variable that I define in my Jenkinsfile with the shell script?
In below scenario , I am writing the value of the shell variable to a file and reading into a groovy variable. Is there a way to pass data from shell to groovy without writing it to file system?
unstash 'sources'
sh'''
source venv/bin/activate
export AWS_ROLE_ARN=arn:aws:iam::<accountid>:role/<role name>
layer_arn="$(awssume aws lambda list-layer-versions --layer-name dependencies --region us-east-1 --query \"LayerVersions[0].LayerVersionArn\" | tr -d '\"')"
echo $layer_arn > layer_arn
'''
layer_arn = readFile('layer_arn').trim()
You can can shell command line, providing variable value.
sh "some stuff $my_var"
You can defined environment variable and use it within your shell
withEnv(["MY_VAR=${my_var}") {
sh 'some stuff'
}
Regards
I am using the following script to query and fetch data utilizing the solr API and read the generated csv output as a csv file. I want to port this script to run in RShiny server. Any suggestions please?
url <- "curl 'https://[site.orgnaization.com/solr/someFolder/select?q=*:*&rows=1000&wt=csv' -L -u username:password --location-trusted -b cookie-jar.txt > C:/Folder/Filename.csv"
fileConn <- file("C:/Folder/PowerShellFile.ps1")
writeLines(url, fileConn)
close(fileConn)
system2("PowerShell", args = c("-file", "C:\\Folder\\PowerShellFile.ps1"))
DataFromSolr <- data.frame(read.csv("C:/Folder/Filename.csv",header = TRUE, sep = "," , fill = TRUE ,fileEncoding = "UTF-16LE"))
I installed PowerShell from this site [https://github.com/PowerShell/PowerShell/blob/master/docs/installation/linux.md#ubuntu-1604] on the linux server and that solved the problem.
I set up a local OpenCPU single-user server using RStudio. I also create my own R package(Package name: test) which includes only a simple test.R file. The source code is
f1 <- function(x, y) {x+y}
I started the OpenCPU server by typing library(opencpu) in RStudio's console. I got the following print.
Initiating OpenCPU server...
OpenCPU started.
[httpuv] http://localhost:6067/ocpu
OpenCPU single-user server ready.
I was able to run the script by typing curl http://localhost:6067/ocpu/library/test/R/f1 -d "x=33&y=3".
But when I tried to display the R script(test.R) by typing curl http://localhost:6067/ocpu/library/test/R/test.R, it printed
object 'test.R' not found
In call:
get(reqobject, paste("package", reqpackage, sep = ":"), inherits = FALSE)
In addition, It's failed when I ran the test.R script by typing curl http://localhost:6067/ocpu/library/test/R/test.R -X POST -d "x=3&y=4". Could I run the script like that?
Could anyone help with this? Thanks.
When you install the R package, scripts under /R are turned into functions/objects. To read the source of the function, just do on of these:
curl http://localhost:6067/ocpu/library/test/R/f1/print
curl http://localhost:6067/ocpu/library/test/R/f1/ascii