I would like to copy a file from my computer to a remote server via SCP using R.
I have found 2 functions that appear to satisfy this partially.
1.
Using ssh.utils
ssh.utils::cp.remote(path.src="~/myfile.txt",
remote.dest="username#remote",
remote.src="", path.dest="~/temp", verbose=TRUE)
I've noticed that with this method, if I need to enter a password (when remote doesn't have my public key), the function produces an error.
2.
Using RCurl:
RCurl appears to have more robust functionality in the scp() function, but, from what I can tell, it is only for copying a file from a remote to my local machine. I would like to do the opposite.
Is there another way to use these functions or is there another function that would be able to copy a file from my local machine to a remote machine via SCP?
One approach to address the need to enter a password interactively is to use sshpass (see https://stackoverflow.com/a/13955428/6455166) in a call to system, e.g.
system('sshpass -p "password" scp ~/myfile.txt username#remote.com:/some/remote/path')
See the linked answer above for more details, including options to avoid embedding the password in the command.
Related
I have been using R function getURL() to load data on RStudio from a remote FTP server. However, this requires having my username and password visible in the script.
require("RCurl")
getURL("ftp://directory/filename.txt", userpwd="user:pwd")
Is there a way to hide this information?
You could use the Keyring package.
library(keyring)
key_set(service = "curl_page",
username = "joe")
Then enter your password when requested. Then you can retrieve it using:
require("RCurl")
getURL("ftp://directory/filename.txt", userpwd=key_get("curl_page",username = "joe"))
This is something of a guess, as I'm not familiar with R, but the normal way to do this (in any language) is to pass the username and password via environment variables that are set from an external source, such as a .env file that is not checked into your source repo, or are passed in from settings handled by your VM's hypervisor (if you have one). That way your credentials never hit your repo and do not appear directly in the source. It's also convenient if you want to run the code in different contexts, such as local, test, stage, production, etc.
This answer looks like a reasonable description of how to do this in R.
I am trying to copy files from my edge node to HDFS using oozie. Many suggested to setup password less ssh to get this done.
Iam unable to login to oozie user as it is a service user.
Is there any other way other than password less ssh.
Thanks in advance.
Other than password less ssh there are two more options :
1. My preferred option : Use JSch java library and create a java application which will accept a shell script to be executed as argument. Using the JSch , it will perform ssh on the configured edge node and execute the shell script on the edge node. In the jsch, you can configure, the edgenode username and password. Use 'JCEKS' file to store the password.
Then add a Java Action in Oozie to run the java application created using JSch.
2. Use "/usr/bin/expect" library to create a shell script, which will perform ssh on edgenode and then run the configured shell script. More details are here Use expect in bash script to provide password to SSH command
Inspired by this awesome post on a Git branching model and this one on what a version bumping script actually does, I went about creating my own Git version bumping routine which resulted in a little package called bumpr.
However, I don't like the current way of handling (GitHub) HTTPS credentials. I'm using the solution stated in this post and it works great, but I don't like the fact that I need to store my credentials in plain text in this _netrc file.
So I wondered:
if one could also obfuscate console input when prompting via readline(), scan() or the like in much the same way as when using the Git shell. See code of /R/bump.r at line 454:
input <- readline(paste0("Password for 'https://",
git_user_email, "#github.com': "))
idx <- ifelse(grepl("\\D", input), input, NA)
if (is.na(idx)){
message("Empty password")
message("Exiting")
return(character())
}
git_https_password <- input
how RStudio realizes that a "Insert credentials" box pops up when pushing to a remote Git repository and how they obfuscate the password entry.
if file _netrc is something closely related to the GitHub API or if this works for HTTPS requests in general
Git has a mechanism to store, cache or prompt for credentials. Please read http://git-scm.com/docs/gitcredentials.
Within a script, you can use the git credential command to access it: http://git-scm.com/docs/git-credential
Is is possible to have R connect to gmail's POP server and read/download the messages in a specific folder of mine? I have been storing emails and would like to go back and start to analyze subject lines, etc.
Basically, I need a way to export a folder in my gmail account and I would like to do this pro grammatically if it all possible.
Thanks in advance!
I am not sure that this can be done via a single command. Maybe there is a package out there, which I am not aware of that can accomplish that, but as long as you do not run into that maybe the following process would be a solution ...
Consider got-your-back (http://code.google.com/p/got-your-back/wiki/GettingStarted#Step_4%3a_Performing_A_Backup) which "is a command line tool that backs up and restores your Gmail account".
You can invoke it like this (given that python is available on your machine):
python gyb.py --email foo#bar.com --search "from:pip#pop.com" --folder "mail_from_pip"
After completion you'll find all the emails matching the --search in the specified --folder, along with a sqlite database. (posted by dukedave, Dec 4 '11)
So depending on your OS you should be able to invoke the above command from within R and then access the downloaded mails in the respective folder.
GotYourBack is a good backup utility, but for downloading metadata for analysis, you might want something that doesn't first require you to fetch the entire content of all your email.
I've recently used the gmailr package to do a similar analysis.
I want to copy files from local machine to remote server through SCP command. I was able to copy files through Cygwin. Now I want to achieve this thru Qt. How do I invoke 'SCP' command? If I make use of QProcess, then how will I get to know about password prompt?
As fara as I know, you will hit the same issue with scp prompting for the password whichever way you try to call the command (even if you try writing a bash script that calls scp, for instance)
Possible solution I'm aware of includes:
create a public/private key and distribute them so that you do not need to be prompted with the password. An interesting paper on the topic is here
create an expect script and invoke it with QProcess
Require your users to use public keys and your problem is solved: https://hkn.eecs.berkeley.edu/~dhsu/ssh_public_key_howto.html
scp, ssh, sftp, etc. deliberately make it hard to take a password other than directly from the user. Specifically, things like expect will not work. And if they did, they wouldn't be secure.