Using R and paws: How to set credentials using profile in config file? - r

I use SSO and a profile as defined in ~/.aws/config (MacOS) to access AWS services, for instance:
aws s3 ls --profile myprofilename
I would like to access AWS services from within R, using the paws() package. In order to do this, I need to set my credentials in the R code. I want to do this through accessing the profile in the ~/.aws/config file (as opposed to listing access keys in the code), but I haven't been able to figure out how to do this.
I looked at the extensive documentation here, but it doesn't seem to cover my use case.
The best I've been able to come up with is:
x = s3(config = list(credentials = list(profile = "myprofilename")))
x$list_objects()
... which throws an error: "Error in f(): No credentials provided", suggesting that the first line of code above does not connect to my profile as stored in ~/.aws/config.

An alternative is to generate a user/key with programmatic access to your S3 data. Then, assuming that ~/.aws/env contains the values of the generated key:
AWS_ACCESS_KEY_ID=abc
AWS_SECRET_ACCESS_KEY=123
AWS_REGION=us-east-1
insert the following line at the beginning of your file:
readRenviron("~/.aws/env")

This AWS blog provides details about how to get the temporary credentials for programatic access. If you can get the credentials and set the appropriate environment variables, then the code should work fine without the profile name.
Or You can also try the following if you can get temporary credentials using aws cli
Check if you can generate temporary credentials
aws sts assume-role --role-arn <value> --role-session-name <some-meaningful-session-name> --profile myprofilename
If you can execute the above successfully, then you can use this method to automate the process of generating credentials before your code runs.
Put the above command in a bash script get-temp-credentials.sh and generate a JSON containing the temporary credentials as per the documentation.
Add a new profile programmatic-access in the ~/.aws/config
[profile programmatic-access]
credential_process = "/path/to/get-temp-credentials.sh"
Finally update the code to use the profile name as programmatic-access

If you have AWS cli credentials set up as a bash profile eg. ~/.aws/config:
[profile myprof]
region=eu-west-2
output=json
.. and credentials eg. ~/.aws/credentials:
[myprof]
aws_access_key_id = XXX
aws_secret_access_key = xxx
.. paws will use these if you add a line to ~/.Renviron:
AWS_PROFILE=myprof

Related

Authentication for Bigquery using bigrquery from an R Markdown document

I am having problems using bigrquery to connect to a GCP service account from within an R Markdown document that I knit. When I attempt from the console, authentication works fine. Both
library(bigrquery)
bq_auth()
and
library(bigrquery)
bq_auth(email="my-service-account-email#myproject.iam.gserviceaccount.com")
launch a browser with a dialog that lets me pick and authenticate using the specified account as expected. But in the R Markdown, any attempt like
options("httr_oob_default" = TRUE)
bq_auth(email="my-service-account-email#myproject.iam.gserviceaccount.com")
or even using the full list like this
bq_auth(
email = "my-service-account-email#myproject.iam.gserviceaccount.com",
path = NULL,
scopes = c("https://www.googleapis.com/auth/bigquery"),
cache = gargle::gargle_oauth_cache(),
use_oob = gargle::gargle_oob_default(),
token = NULL
)
leads to the error
Error: Can't get Google credentials.
Are you running bigrquery in a non-interactive session? Consider:
* Call `bq_auth()` directly with all necessary specifics.
Can anyone see what I am missing? Thanks in advance.
You can download the JSON file of your Google Cloud service account, then use it as a path that the “bq_auth” function can recognize. Here's the steps:
Google Cloud Console (console.cloud.google.com)
Navigation Menu
IAM & Admin Service
Accounts
Create Service Account (create one)
Create Key, and save to "/path/to/jsonfilename.json"
Authenticate in your R Markdown code: bigrquery::bq_auth(path = "/path/to/jsonfilename.json")
Note: you'll need to make sure to set the service account to have access to BigQuery. I set mine to "BigQuery Admin" and it worked, but that might be too broad
Borrowed this answer from Elaine See's post on medium: https://medium.com/#elaine.yl.see/easiest-way-to-use-bigquery-in-r-8af466cd55ca

While configuring BPS DB in wso2 is 5.9.0 , which scripts do i have to import in MySQL?

I am following this document-https://is.docs.wso2.com/en/5.9.0/setup/changing-datasource-bpsds/
deployment.toml Configurations.
[bps_database.config]
url = "jdbc:mysql://localhost:3306/IAMtest?useSSL=false"
username = "root"
password = "root"
driver = "com.mysql.jdbc.Driver"
Executing database scripts.
Navigate to <IS-HOME>/dbscripts. Execute the scripts in the following files, against the database created.
<IS-HOME>/dbscripts/bps/bpel/create/mysql.sql
<IS-HOME>/dbscripts/bps/bpel/drop/mysql-drop.sql
<IS-HOME>/dbscripts/bps/bpel/truncate/mysql-truncate.sql
Now create/mysql.sql creates table and the rest two file are responsible for deleting and trucating the same table..............what do i do?????????
Can anyone also tell the use case of BPS datasource??????
Please Help...........
You should only change your bps database if you have a requirement of using the workflow feature[1] in the wso2 identity server. It is mentioned in this documentation https://is.docs.wso2.com/en/5.9.0/setup/changing-to-mysql/
The document supposed to menstion the related db script. But it seems like mis leading. As it has requested to execute all three scripts. if you are using the workflow feature just use the
/dbscripts/bps/bpel/create/mysql.sql
script to create tables in you mysql database.
[1]. https://is.docs.wso2.com/en/5.9.0/learn/workflow-management/

How to knit dynamic reports with Google Analytics (rga)

I'm using rga to get some data from Google Analytics. From the repo:
The principle of this package is to create an instance of the API Authentication, which is a S4/5-class (utilizing the setRefClass). This instance then contains all the functions needed to extract data, and all the data needed for the authentication and reauthentication. The class is in essence self sustaining.
The package creates and saves a local instance using:
rga.open(instance="ga", where="~/ga.rga")
When I try to knit, however, I get an error that the ga object (what would be the instance) is not found. The code works when I run the chunks in RStudio, however—I believe the error is related to this aspect:
[The command above] will check if the instance is already created, and if it is, it'll prepare the token. If the instance is not created [...] it will redirect the client to a browser for authentication with Google.
My guess is that knitr can't perform that last step and so, the object is never created.
How can I make this work? I'm thinking that there might be a way to load the local ga.rga file to bypass browser authentication.
You can bypass browser authentication by passing the client id and client secret key that you can get it from Google API console. Saving a local auth file in the dev env is always risky. You can try this code, this uses Google API and also saves the local instance -
rga.open(instance = "ga",
client.id = "<contains apps.googleusercontent.com>",
client.secret =<your secret key>, where ="~/ga.rga" )
Also ensure that desktop option setting is enabled in Google API console

Running AWS commands from commandline on a ShellCommandActivity

My original problem was that I want to increase my DynamoDB write throughput before I run the pipeline, and then decrease it when I'm done uploading (doing it max once a day, so I'm fine with the decreasing limitations).
They only way I found to do it is through a shell script that will issue the API commands to alter the throughput. How does it work with my AMI access_key and secret_key when it's a resource that pipeline creates for me? (I can't log in to set the ~/.aws/config file and don't really want to create an AMI just for this).
Should I write the script in bash? can I use ruby/python AWS SDK packages for example? (I prefer the latter..)
How do I pass my credentials to the script? do I have runtime variables (like #startedDate) that I can pass as arguments to the activity with my key and secret? Do I have any other way to authenticate with either the commandline tools or the SDK package?
If there is another way to solve my original problem - please let me know. I've only got to the ShellActivity solution because I couldn't find anything else in documentations/forums.
Thanks!
OK. found it - http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-concepts-roles.html
The resourceRole in the default object in your pipeline will be the one assigned to resources (Ec2Resource) that are created as a part of the pipeline activation.
The default one in configured to have all your permissions and AWS commandline and SDK packages are automatically looking for those credentials so no need to update ~/.aws/config of pass credentials manually.

cloudmonkey with crontab?

i have a Cloudstack 4.2.1 here and would like my VMs to boot from time and shutdown at a scheduled time.
Hence i was thinking if i could integrate Cloudmonkey with CronTab together.
Firstly by creating a Cloudmonkey Script or API call then using crontab to run it at a specific time.
However i have problems creating a Cloudmonkey script/API call...
i haved googled and found this link
http://dlafferty.blogspot.sg/2013/07/using-cloudmonkey-to-automate.html
and had a result of
apiresult=cloudmonkey api stop virtualmachine id="'e10bdf21-2d5c-4277-9d8d-791b82b9e3be'"
unfortunately when i entered this command, nothing happened. If anyone could have an alternative suggestion or rather my API call command is wrong, please correct me and help
Thank you.
CloudMonkey requires some setup before it works (e.g. setting your API key).
Check [1] for the documentation for CloudMoney and follow through the Usage section to setup your environment.
Once your setup is complete and you can interact with CloudStack via CloudMonkey, you should take into account that the VM ids might change, so before you issue a command for a VM, you should first find the correct id, by listing the VMs and picking the right one.
Also, if you run into trouble, post the relevant log from CLoudStack management server (typically in /var/log/cloudstack/management/management-server.log).
[1] - https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI
Edit: If you have a working connection via CloudMonkey to CloudStack, you need to configure CloudMonkey in the same way in your shell script. For instance when you configured CloudMonkey you probably set a host, a port and your api and secret keys. So for your scrip to work you need to provide the same configuration to CloudMonkey prior to issuing the commands. My best guess is to use the -c option and provide a config file to set all the relevant parameters (e.g. api an secret key). cloudmonkey -c CONFIG_FILE ....
Edit2: You don't actually need to re-configure cloudmonkey in your script because it will remember your config from the interactive session. I would still advise you to do it, because your script gets more reliable. I've just made an example script like this:
#! /bin/bash
result=$(cloudmonkey list users)
echo $result
Result:
> ./tmp.sh
count = 1 user: id = 678e3a24-082c-11e4-86de-acbdb2423647 account = admin accountid = 678dffe6-082c-11e4-86de-acbdb2423647 accounttype = 1 apikey = T6sDBIpytyJ4_PMgNXYi8YgjMtwTiiDjijbXNB1J78EAZq2foKhCoGKjgJnej5tMaHM0LUvejgTddkhVU63wdw created = 2014-07-10T16:19:13+0200 domain = ROOT domainid = 678dd7b4-082c-11e4-86de-acbdb2423647 email = admin#mailprovider.com firstname = Admin iscallerchilddomain = False isdefault = True lastname = User secretkey = dzOPRecI5vvEVK7Vie2D0tDsQGXunUnpIAczbXnPI3sfMwQ-upWL_bPOisEYg4C-nXi-ldQno2KVZbVR-5NmVw state = enabled username = admin
Maybe you forgot to echothe result?

Resources