I have been using R function getURL() to load data on RStudio from a remote FTP server. However, this requires having my username and password visible in the script.
require("RCurl")
getURL("ftp://directory/filename.txt", userpwd="user:pwd")
Is there a way to hide this information?
You could use the Keyring package.
library(keyring)
key_set(service = "curl_page",
username = "joe")
Then enter your password when requested. Then you can retrieve it using:
require("RCurl")
getURL("ftp://directory/filename.txt", userpwd=key_get("curl_page",username = "joe"))
This is something of a guess, as I'm not familiar with R, but the normal way to do this (in any language) is to pass the username and password via environment variables that are set from an external source, such as a .env file that is not checked into your source repo, or are passed in from settings handled by your VM's hypervisor (if you have one). That way your credentials never hit your repo and do not appear directly in the source. It's also convenient if you want to run the code in different contexts, such as local, test, stage, production, etc.
This answer looks like a reasonable description of how to do this in R.
Related
I am using a group context to configure the db connection. The password of the db has a password type. When deploying the job, the password is automatically encrypted in the default.properties under the contexts folder.
What if i want to change the password without using the studio (on a client environment)? what can i use to encrypt the new password?
I was able to do it by creating a separate encryption job with a tjava component and the following code:
System.out.println(routines.system.PasswordEncryptUtil.encryptPassword(context.Password));
where context.Password is an input context variable of type String. When running the job, the user is prompted to enter a password and then the encrypted Talend password will be printed. It will have the following format: enc:routine.encryption.key.v1:[encryptedPassword] The routine encryption key can be modified if needed by following this link: https://help.talend.com/r/en-US/8.0/installation-guide-data-integration-windows/rotating-encryption-keys-in-talend-studio
There's actually a few ways for this:
myJob.sh --context_param myPassword=pass123
this unfortunately can be seen by anyone via ps / task manager.
You can also edit the contexts/contextName.properties file and change the context parameters there. This way the context can only be seen if you have access to the file.
Theoretically both should be able to accept the cleartext/encrypted password.
Implicit context load feature can also be used to load contexts: https://help.talend.com/r/en-US/8.0/data-integration-job-examples/creating-job-and-defining-context-variables
In the R script, when I try to send the email with the following codes below. It asks that the gmailr package is requesting access to your Google account. Select a pre-authorised account or enter '0' to obtain a new token. Press Esc/Ctrl + C to abort.
1: email1#gmail.com
without manually entering 1 in the console, how can my R script automatically select my pre-authorised account and sent an email accordingly?
library(gmailr)
gm_auth_configure(path="C:/Users/Google Drive/email.json")
my_email_message <- gm_mime() %>%
gm_to("email1#gmail.com") %>%
gm_from("email1#gmail.com") %>%
gm_subject("My test message")
gm_send_message(my_email_message)
This is the unattended / non-interactive authentication problem. I will try to give the rundown of the process as it worked for me - and the problem, exactly like yours, went away. As it states in gmailr/readme - you download json credentials, authenticate once interactively and copy creds to wherever you like. Credentials you can get via python quickstart, or even better - by simply creating a project on https://console.developers.google.com, adding gmail API to it, then creating OAuth credentials for a desktop app. The benefit of the latter approach is you will know exactly where all components are and will be able to repeat as many times as you want. I created a separate google e-mail address for this purpose. You will then download OAuth "client-secret" .json file into your project directory and call it credentials.json (or any other json name you like). Then you will once authenticate interactively running below commands from Rstudio when you are in your project directory:
gm_auth_configure(path = "credentials.json")
gm_auth(email = TRUE, cache = ".secret")
A webpage will pop up with scary messages, but you will agree to all and from then on you will be using cache. Cache .secret sub-directory that you just created inside your project (and you can give whatever name you wish to the cache directory) is portable - you can copy that alongside your credentials.json over to your shiny-server. It is convenient that all is contained in your project directory. You will need a few lines in your code after that - they should precede the command gm_send_message(your_email_prepared_with_gm_mime) and no more interactive authentication is needed no matter which computer you have copied your project to as long as it has gmailr and gargle (which is a gmailr dependency) installed in R on your server:
gm_auth_configure(path = "credentials.json")
options(
gargle_oauth_cache = ".secret",
gargle_oauth_email = "email_address_used_for_creds#gmail.com"
)
gm_auth(email = "email_address_used_for_creds#gmail.com")
# then compose your e-mail and send it
the last command allows to avoid dialogue for which account to use. This sometimes pops up on first use.
gmailr Readme explains it well; my explanation is an encouragement to read it again, if you get stuck. You can read also gmailr reference at https://gmailr.r-lib.org/index.html - it is pretty good. But my guess is - if you have followed the process here you won't even need that.
Note on cache: Default gargle (this is what makes authentication for gmailr happen) cache directory is in some hidden subdirectory of your home directory - so it is specific to you on that computer. However if you set it to be a subdirectory to your R project, the whole OAuth process becomes portable. Just copy your project directory were you want and the OAuth credential pair - the json file and OAuth token(s) in the cache will follow along. Tokens are gzipped binary files that gmail creates cryptographically and deposits in the cache during the "authentication dance". One address paired to one G-project gives one token. One probably could use multiple addresses and google projects in one R project, but I so far have yet to see the need for that.
Just add the "from e-mail address" with gm_auth(email = "email1#gmail.com")
library(gmailr)
gm_auth_configure(path="C:/Users/Google Drive/email.json")
gm_auth(email = "email1#gmail.com")
my_email_message <- gm_mime() %>%
gm_to("email1#gmail.com") %>%
gm_from("email1#gmail.com") %>%
gm_subject("My test message")
gm_send_message(my_email_message)```
I would like to copy a file from my computer to a remote server via SCP using R.
I have found 2 functions that appear to satisfy this partially.
1.
Using ssh.utils
ssh.utils::cp.remote(path.src="~/myfile.txt",
remote.dest="username#remote",
remote.src="", path.dest="~/temp", verbose=TRUE)
I've noticed that with this method, if I need to enter a password (when remote doesn't have my public key), the function produces an error.
2.
Using RCurl:
RCurl appears to have more robust functionality in the scp() function, but, from what I can tell, it is only for copying a file from a remote to my local machine. I would like to do the opposite.
Is there another way to use these functions or is there another function that would be able to copy a file from my local machine to a remote machine via SCP?
One approach to address the need to enter a password interactively is to use sshpass (see https://stackoverflow.com/a/13955428/6455166) in a call to system, e.g.
system('sshpass -p "password" scp ~/myfile.txt username#remote.com:/some/remote/path')
See the linked answer above for more details, including options to avoid embedding the password in the command.
Inspired by this awesome post on a Git branching model and this one on what a version bumping script actually does, I went about creating my own Git version bumping routine which resulted in a little package called bumpr.
However, I don't like the current way of handling (GitHub) HTTPS credentials. I'm using the solution stated in this post and it works great, but I don't like the fact that I need to store my credentials in plain text in this _netrc file.
So I wondered:
if one could also obfuscate console input when prompting via readline(), scan() or the like in much the same way as when using the Git shell. See code of /R/bump.r at line 454:
input <- readline(paste0("Password for 'https://",
git_user_email, "#github.com': "))
idx <- ifelse(grepl("\\D", input), input, NA)
if (is.na(idx)){
message("Empty password")
message("Exiting")
return(character())
}
git_https_password <- input
how RStudio realizes that a "Insert credentials" box pops up when pushing to a remote Git repository and how they obfuscate the password entry.
if file _netrc is something closely related to the GitHub API or if this works for HTTPS requests in general
Git has a mechanism to store, cache or prompt for credentials. Please read http://git-scm.com/docs/gitcredentials.
Within a script, you can use the git credential command to access it: http://git-scm.com/docs/git-credential
Is is possible to have R connect to gmail's POP server and read/download the messages in a specific folder of mine? I have been storing emails and would like to go back and start to analyze subject lines, etc.
Basically, I need a way to export a folder in my gmail account and I would like to do this pro grammatically if it all possible.
Thanks in advance!
I am not sure that this can be done via a single command. Maybe there is a package out there, which I am not aware of that can accomplish that, but as long as you do not run into that maybe the following process would be a solution ...
Consider got-your-back (http://code.google.com/p/got-your-back/wiki/GettingStarted#Step_4%3a_Performing_A_Backup) which "is a command line tool that backs up and restores your Gmail account".
You can invoke it like this (given that python is available on your machine):
python gyb.py --email foo#bar.com --search "from:pip#pop.com" --folder "mail_from_pip"
After completion you'll find all the emails matching the --search in the specified --folder, along with a sqlite database. (posted by dukedave, Dec 4 '11)
So depending on your OS you should be able to invoke the above command from within R and then access the downloaded mails in the respective folder.
GotYourBack is a good backup utility, but for downloading metadata for analysis, you might want something that doesn't first require you to fetch the entire content of all your email.
I've recently used the gmailr package to do a similar analysis.