Inspired by this awesome post on a Git branching model and this one on what a version bumping script actually does, I went about creating my own Git version bumping routine which resulted in a little package called bumpr.
However, I don't like the current way of handling (GitHub) HTTPS credentials. I'm using the solution stated in this post and it works great, but I don't like the fact that I need to store my credentials in plain text in this _netrc file.
So I wondered:
if one could also obfuscate console input when prompting via readline(), scan() or the like in much the same way as when using the Git shell. See code of /R/bump.r at line 454:
input <- readline(paste0("Password for 'https://",
git_user_email, "#github.com': "))
idx <- ifelse(grepl("\\D", input), input, NA)
if (is.na(idx)){
message("Empty password")
message("Exiting")
return(character())
}
git_https_password <- input
how RStudio realizes that a "Insert credentials" box pops up when pushing to a remote Git repository and how they obfuscate the password entry.
if file _netrc is something closely related to the GitHub API or if this works for HTTPS requests in general
Git has a mechanism to store, cache or prompt for credentials. Please read http://git-scm.com/docs/gitcredentials.
Within a script, you can use the git credential command to access it: http://git-scm.com/docs/git-credential
Related
In the R script, when I try to send the email with the following codes below. It asks that the gmailr package is requesting access to your Google account. Select a pre-authorised account or enter '0' to obtain a new token. Press Esc/Ctrl + C to abort.
1: email1#gmail.com
without manually entering 1 in the console, how can my R script automatically select my pre-authorised account and sent an email accordingly?
library(gmailr)
gm_auth_configure(path="C:/Users/Google Drive/email.json")
my_email_message <- gm_mime() %>%
gm_to("email1#gmail.com") %>%
gm_from("email1#gmail.com") %>%
gm_subject("My test message")
gm_send_message(my_email_message)
This is the unattended / non-interactive authentication problem. I will try to give the rundown of the process as it worked for me - and the problem, exactly like yours, went away. As it states in gmailr/readme - you download json credentials, authenticate once interactively and copy creds to wherever you like. Credentials you can get via python quickstart, or even better - by simply creating a project on https://console.developers.google.com, adding gmail API to it, then creating OAuth credentials for a desktop app. The benefit of the latter approach is you will know exactly where all components are and will be able to repeat as many times as you want. I created a separate google e-mail address for this purpose. You will then download OAuth "client-secret" .json file into your project directory and call it credentials.json (or any other json name you like). Then you will once authenticate interactively running below commands from Rstudio when you are in your project directory:
gm_auth_configure(path = "credentials.json")
gm_auth(email = TRUE, cache = ".secret")
A webpage will pop up with scary messages, but you will agree to all and from then on you will be using cache. Cache .secret sub-directory that you just created inside your project (and you can give whatever name you wish to the cache directory) is portable - you can copy that alongside your credentials.json over to your shiny-server. It is convenient that all is contained in your project directory. You will need a few lines in your code after that - they should precede the command gm_send_message(your_email_prepared_with_gm_mime) and no more interactive authentication is needed no matter which computer you have copied your project to as long as it has gmailr and gargle (which is a gmailr dependency) installed in R on your server:
gm_auth_configure(path = "credentials.json")
options(
gargle_oauth_cache = ".secret",
gargle_oauth_email = "email_address_used_for_creds#gmail.com"
)
gm_auth(email = "email_address_used_for_creds#gmail.com")
# then compose your e-mail and send it
the last command allows to avoid dialogue for which account to use. This sometimes pops up on first use.
gmailr Readme explains it well; my explanation is an encouragement to read it again, if you get stuck. You can read also gmailr reference at https://gmailr.r-lib.org/index.html - it is pretty good. But my guess is - if you have followed the process here you won't even need that.
Note on cache: Default gargle (this is what makes authentication for gmailr happen) cache directory is in some hidden subdirectory of your home directory - so it is specific to you on that computer. However if you set it to be a subdirectory to your R project, the whole OAuth process becomes portable. Just copy your project directory were you want and the OAuth credential pair - the json file and OAuth token(s) in the cache will follow along. Tokens are gzipped binary files that gmail creates cryptographically and deposits in the cache during the "authentication dance". One address paired to one G-project gives one token. One probably could use multiple addresses and google projects in one R project, but I so far have yet to see the need for that.
Just add the "from e-mail address" with gm_auth(email = "email1#gmail.com")
library(gmailr)
gm_auth_configure(path="C:/Users/Google Drive/email.json")
gm_auth(email = "email1#gmail.com")
my_email_message <- gm_mime() %>%
gm_to("email1#gmail.com") %>%
gm_from("email1#gmail.com") %>%
gm_subject("My test message")
gm_send_message(my_email_message)```
I have been using R function getURL() to load data on RStudio from a remote FTP server. However, this requires having my username and password visible in the script.
require("RCurl")
getURL("ftp://directory/filename.txt", userpwd="user:pwd")
Is there a way to hide this information?
You could use the Keyring package.
library(keyring)
key_set(service = "curl_page",
username = "joe")
Then enter your password when requested. Then you can retrieve it using:
require("RCurl")
getURL("ftp://directory/filename.txt", userpwd=key_get("curl_page",username = "joe"))
This is something of a guess, as I'm not familiar with R, but the normal way to do this (in any language) is to pass the username and password via environment variables that are set from an external source, such as a .env file that is not checked into your source repo, or are passed in from settings handled by your VM's hypervisor (if you have one). That way your credentials never hit your repo and do not appear directly in the source. It's also convenient if you want to run the code in different contexts, such as local, test, stage, production, etc.
This answer looks like a reasonable description of how to do this in R.
I would like to copy a file from my computer to a remote server via SCP using R.
I have found 2 functions that appear to satisfy this partially.
1.
Using ssh.utils
ssh.utils::cp.remote(path.src="~/myfile.txt",
remote.dest="username#remote",
remote.src="", path.dest="~/temp", verbose=TRUE)
I've noticed that with this method, if I need to enter a password (when remote doesn't have my public key), the function produces an error.
2.
Using RCurl:
RCurl appears to have more robust functionality in the scp() function, but, from what I can tell, it is only for copying a file from a remote to my local machine. I would like to do the opposite.
Is there another way to use these functions or is there another function that would be able to copy a file from my local machine to a remote machine via SCP?
One approach to address the need to enter a password interactively is to use sshpass (see https://stackoverflow.com/a/13955428/6455166) in a call to system, e.g.
system('sshpass -p "password" scp ~/myfile.txt username#remote.com:/some/remote/path')
See the linked answer above for more details, including options to avoid embedding the password in the command.
I'm looking to call BigQuery from R Studio, installed on a Google Compute Engine.
I have the bq python tool installed on the instance, and I was hoping to use its service accounts and system() to get R to call bq command line tool and so get the data.
However, I run into authentication problems, where it asks for a browser key. I'm pretty sure there is no need to get the key due to the service account, but I don't know how to construct the authetication from with R (it runs on RStudio, so will have multiple users)
I can get an authetication token like this:
library(RCurl)
library(RJSONIO)
metadata <- getURL('http://metadata/computeMetadata/v1beta1/instance/service-accounts/default/token')
tokendata <- fromJSON(metadata)
tokendata$$access_token
But how do I then use this to generate a .bigqueryrc token? Its the lack of this that triggers the authetication attempt.
This works ok:
system('/usr/local/bin/bq')
showing me bq is installed ok.
But when I try something like:
system('/usr/local/bin/bq ls')
I get this:
Welcome to BigQuery! This script will walk you through the process of initializing your .bigqueryrc configuration file.
First, we need to set up your credentials if they do not already exist.
******************************************************************
** No OAuth2 credentials found, beginning authorization process **
******************************************************************
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=XXXXXXXX.apps.googleusercontent.com&access_type=offline
Enter verification code: You have encountered a bug in the BigQuery CLI. Google engineers monitor and answer questions on Stack Overflow, with the tag google-bigquery: http://stackoverflow.com/questions/ask?tags=google-bigquery
etc.
Edit:
I have managed to get bq functioning from RStudio system() commands, by skipping the authetication by logging in to the terminal as the user using RStudio, autheticating there by signing in via the browser, then logging back to RStudio and calling system("bq ls") etc..so this is enough to get me going :)
However, I would still prefer it if BQ can be autheticated within RStudio itself, as many users may log in and I'll need to autheticate via terminal for all of them. And from the service account documentation, and the fact I can get an authetication token, hints at this being easier.
For the time being, you need to run 'bq init' from the command line to set up your credentials prior to invoking bq from a script in GCE. However, the next release of bq will include support for GCE service accounts via a new --use_gce_service_account flag.
Is is possible to have R connect to gmail's POP server and read/download the messages in a specific folder of mine? I have been storing emails and would like to go back and start to analyze subject lines, etc.
Basically, I need a way to export a folder in my gmail account and I would like to do this pro grammatically if it all possible.
Thanks in advance!
I am not sure that this can be done via a single command. Maybe there is a package out there, which I am not aware of that can accomplish that, but as long as you do not run into that maybe the following process would be a solution ...
Consider got-your-back (http://code.google.com/p/got-your-back/wiki/GettingStarted#Step_4%3a_Performing_A_Backup) which "is a command line tool that backs up and restores your Gmail account".
You can invoke it like this (given that python is available on your machine):
python gyb.py --email foo#bar.com --search "from:pip#pop.com" --folder "mail_from_pip"
After completion you'll find all the emails matching the --search in the specified --folder, along with a sqlite database. (posted by dukedave, Dec 4 '11)
So depending on your OS you should be able to invoke the above command from within R and then access the downloaded mails in the respective folder.
GotYourBack is a good backup utility, but for downloading metadata for analysis, you might want something that doesn't first require you to fetch the entire content of all your email.
I've recently used the gmailr package to do a similar analysis.