I am attempting to connect to an SFTP site to pull data. It used to work, but for some reason, it stopped working a couple of weeks ago. The owners of the SFTP say nothing has changed on their end, and I can pull data easily without error using WinSCP.
protocol <- "sftp"
server <- "sftp.xxxx.net"
userpwd <- "user:password"
file <- "/public/bpus_dailytx.csv"
url <- paste0(protocol, "://", server, file)
data <- getURL(url = url, userpwd=userpwd, verbose = TRUE)
When I run this, I now get the following info:
* Trying xxx.xx.xx.xxx...
* Connected to sftp.xxxx.net (xxx.xx.xx.xxx) port 22 (#0)
* SSH MD5 fingerprint: 34rh3ie93hhr39hhdik3
* SSH authentication methods available: publickey,keyboard-interactive
* Using SSH public key file '(nil)'
* Using SSH private key file ''
* SSH public key authentication failed: Unable to extract public key from private key file: Unable to open private key file
* No identity would match
* Authentication failure
* Closing connection 0
Error in function (type, msg, asError = TRUE) : Authentication failure
It connects OK but then the authentication fails. Any ideas what could be going on here? Again, this code used to work but something has changed. What are some other ways I can attempt to pull the data other than this?
Edit: WinSCP screenshots:
It is difficult to say why your code stopped working because we do not have enough information about configurations (both on your machine and on the server) when it was working.
Because your keyfile was not in the proper format for RCurl, my leading hypothesis is that although the server folks said nothing changed on their end, I think they removed the password authentication option. That is because your code attempts password authentication only. If password authentication were still available, the one line in your output would look something like this:
SSH authentication methods available: publickey,password,keyboard-interactive
It is, as you noted, now:
SSH authentication methods available: publickey,keyboard-interactive
Therefore, the solution here was to convert your keyfile from PuTTY to OpenSSH format using PuTTYgen and then use the following RCurl code pointing to your new keyfile:
protocol <- "sftp"
server <- "sftp.xxxx.net"
file <- "/public/bpus_dailytx.csv"
url <- paste0(protocol, "://", server, file)
keypasswd <- "your_keypasswd"
ssh.private.keyfile = "your_path_to_keyfile"
username <- "your_username"
data <- getURL(url = url, keypasswd = keypasswd, ssh.private.keyfile = ssh.private.keyfile, username = username, verbose = TRUE)
And I will add a special thanks to #Tensibai for the assistance. It would have taken me way longer to arrive at this solution without their insight with the keyfile format.
Related
I'm writing a R package for the first time. Its called aactr and is hosted on my GitHub: https://github.com/jasonbaik94/aactr
Right now, the package only has one function, aact_connect:
aact_connect <- function(user, password) {
drv <- DBI::dbDriver('PostgreSQL')
con <- DBI::dbConnect(drv,
dbname="aact",
host="aact-db.ctti-clinicaltrials.org",
port=5432,
user=user,
password=password)
}
My concern is that users of my package wouldn't like to input their username and password in their R script for privacy reasons.
What would be a nice workaround to ensure user privacy?
One thought I had: When the user types aact_connect(), a window pops up where the user can input username and password and press Enter, after which the connection would be made. Also, for those without a username or password, I'd put a parameter, init_connection = TRUE, after which this sign up page would load: https://aact.ctti-clinicaltrials.org/users/sign_up
Any others suggestions are greatly welcome!
There are many ways to securely handle database credentials in scripts. See few examples below:
Use configuration files such as yaml saves securely on user's machines for R to read into needed variables. See #Spacedman's answer.
config.yaml
db:
host : localhost
port : 5432
name : mypgdb
user : pg_useR
pwd : ***
R
library(yaml)
config = yaml.load_file("/path/to/config.yml")
dbConnect(drv, host = config$db$host, port = config$db$port,
dbname = config$db$name,
user = config$db$user, password = config$db$pwd)
Use environmental variables that are linked to user or system profiles:
db_creds <- Sys.getenv(c("DB_HOST", "DB_PORT", "DB_NAME", "DB_USER", "DB_PWD"))
con <- DBI::dbConnect(drv,
dbname = db_creds[['DB_NAME']],
host = db_creds[['DB_HOST']],
port = db_creds[['DB_PORT,']],
user = db_creds[['DB_USER']],
password = db_creds[['DB_PWD']])
Use DSNs that maintains connection parameters but requires an ODBC connection which can be run with the generalized odbc package (part of same DBI family as RPostgreSQL). Postgres maintains up-to-date odbc drivers for most operating systems. See R-bloggers post.
odbc.ini
[myPG_DSN]
Driver = PostgreSQL Unicode
Database = mypg_db
Servername = localhost
UserName = pg_useR
Password = ***
R
db <- dbConnect(odbc::odbc(), "myPG_DSN")
I'm trying to connect to a SFTP server. It is an encrypted server that uses FIPS mode. I am able to connect and perform file transfer through WinSCP, FileZilla, and through bash sftp commands with no problems.
But, I cannot, access this same SFTP from R using RCurl (R version=3.3.2, RCurl version=1.95-4.10, windows 10). This is what my code looks like and the error message that is being produced:
RCurl::ftpUpload(what="path/to/my/local/file.ext",
to = "sftp://my.eftp.server:portNumber/path/to/my/file.ext",
userpwd = "user:password",
.opts=curlOptions(verbose=TRUE))
The error message is:
* Trying ###.###.###.##...
* Connected to my.eftp.server (###.###.###.##) port ## (#0)
* Failure establishing ssh session
* Closing connection 0
Error in function (type, msg, asError = TRUE) :
Failure establishing ssh session
Any help would be wonderful. I've referenced the following with no luck:
sftp with R - sftp not a protocol with RCurl,
Using RCurl with SFTP
and the RCurl documentation (and other links).
I've also tried:
RCurl::ftpUpload(what="path/to/my/local/file.ext",
to = "sftp://user:password#my.eftp.server:portNumber/path/to/my/file.ext",
.opts=curlOptions(verbose=TRUE))
Also, I've checked my curlVersion()$protocols and scp and sftp protocols are listed (with the others).
I have acces to an oneway export function to a public company database through Elastic Search. I have problems connection to it from R and the elastic package.
I have server name(URL), username and password, but I don't have any port number. They describe it as a rest API. Do I have to use the elastic package or is there an easier way around it. The only information I have to the database is: http://distribution.virk.dk/cvr-permanent/virksomhed/_search?.
Host="Distribution.virk.dk"
index="cvr-permanent"
type="virksomhed"
The above link works with HTTR, but I wish to use elastic for automation purposes, when making a large request of data.
so my connect looks like
host = "distribution.virk.dk"
port = ''
path = ''
schema = "http"
user = "user_name"
pass = "secret"
connect(es_host = host,es_user=user, transport=schema, port=port, es_pwd = pass)
Even though I set port to blank it returns 9200.
If I try to use Search
>Search(index="cvr-permanent", type="virksomhed", q='"cvrNummer":"33647093"', size=10)
Error in curl::curl_fetch_memory(url, handle = handle) :
Failed to connect to distribution.virk.dk port 9200: Timed out
(elastic maintainer here)
You should be able to pass in httr::authenticate() to elastic::Search and other functions from the pkg, e.g,.
x <- Search(config = c(httr::verbose(), authenticate("foo", "bar")))
You should see the Authorization: Basic XXXXXX header in the request headers
does that work?
I need to get config file from the Linux server that is authenticated using ssh keys. I am stuck with "Callback returned error" message.
conf = scp(host="10.10.10.10", path="/home/admin/codebase/config.txt",
user="admin", keypasswd = "", verbose=TRUE,
key=c("C:/echinn/.ssh/my_public_key", "C:/echinn/.ssh/my_private_key"))
I get the following output
* Trying 10.10.10.10...
* Connected to 10.10.10.10 (10.10.10.10) port 22 (#0)
* SSH MD5 fingerprint: 8fa4562037d2f1e68c7ff419f9dc7656
* SSH authentication methods available: publickey,gssapi-keyex,gssapi-with-mic
* Using SSH public key file 'C:/echinn/.ssh/my_public_key'
* Using SSH private key file 'C:/echinn/.ssh/my_private_key'
* SSH public key authentication failed: Callback returned error
* Failure connecting to agent
* Authentication failure
* Closing connection 0
Error in function (type, msg, asError = TRUE) : Authentication failure
I also tried with "getURL()" but ended up with the same error. I am able to successfully connect using putty and WinSCP with (.ppk format of) the same public/private keys.
On the Linux server /var/log/secure I see the following for every execution
sshd[xxxx]: Connection closed by x.x.x.x [preauth]
From the R Documentation for the scp function, it looks like your key does not have a passphrase, so try setting keypasswd = NA or removing the argument altogether.
I'm using the ftpUpload function in the RCurl package to upload files to an sftp file server. I'm having difficulty working out the authentication call.
Below is my call:
ftpUpload(what = "some-file.png",
to = "sftp://some-ftp-server.com:22/path/to/some-file.png",
verbose = TRUE,
userpwd = "my_userid:my_password")
As a result I get:
* About to connect() to some-ftp-server.com port 22 (#0)
* Trying some-ftp-server.com... * connected
* Connected to some-ftp-server.com (some ip address) port 22 (#0)
* SSH authentication methods available: publickey,password
* Using ssh public key file /home/.ssh/id_dsa.pub
* Using ssh private key file /home/.ssh/id_dsa
* SSH public key authentication failed: Unable to open public key file
* Authentication failure
* Closing connection #0
Error in function (type, msg, asError = TRUE) : Authentication failure
I wasn't the one to setup the sftp server, and I'm somewhat of an ssh noob -- apologies. What I do know is that I'm able to login using my_userid and my_password with Filezilla and that the server has an .htaccess and .htpasswd file.
I'm hoping that there is some way to authenticate using ftpUpload with just my userid and password. It seems that password is one of the two available methods, but I can't seem to get ftpUpload to understand that I'd like to use the later alone.
The .htpasswd file seems to contain my_userid:my_password, though the password portion is encrypted. I'm open to loading that in a certain place for ftpUpload to access, but I'm not sure how to point ftpUpload in the right directions.
Finally, I've tried playing around with and looking through the libcurl options listed here: http://www.omegahat.org/RCurl/philosophy.html and more fully explained here: http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
Alas, no luck. Any help appreciated!
It seems a little late, but I am currently trying something similar (public key authentication, though) and got it working!
Here is what I did:
I did set up a public key authentication basically following the instructions at http://www.howtoforge.com/set-up-ssh-with-public-key-authentication-debian-etch
That is, I run on the server the command
ssh-keygen -t rsa -C "e#mail.com" -f "ir_rsa"
to generate a public and a private key. (Just as a sidenote, I first tried to use
puttygen on my local windows machine to create working keys but failed.)
Then I
add the public key to the authorized keys (still on the remote server)
mkdir ~/.ssh
chmod 700 ~/.ssh
cat id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
and copy both the public and private key file on the local machine where
R is running. Then I can upload a file from R using
require(RCurl)
ftpUpload(what = "myfile.txt",
to = "sftp://myusername#my.host.com:22/path/to/myfile",
verbose = TRUE,
.opts = list(
ssh.public.keyfile = "path/to/local/pubkeyfile",
ssh.private.keyfile = "path/to/local/privatekeyfile"
)
)