I have acces to an oneway export function to a public company database through Elastic Search. I have problems connection to it from R and the elastic package.
I have server name(URL), username and password, but I don't have any port number. They describe it as a rest API. Do I have to use the elastic package or is there an easier way around it. The only information I have to the database is: http://distribution.virk.dk/cvr-permanent/virksomhed/_search?.
Host="Distribution.virk.dk"
index="cvr-permanent"
type="virksomhed"
The above link works with HTTR, but I wish to use elastic for automation purposes, when making a large request of data.
so my connect looks like
host = "distribution.virk.dk"
port = ''
path = ''
schema = "http"
user = "user_name"
pass = "secret"
connect(es_host = host,es_user=user, transport=schema, port=port, es_pwd = pass)
Even though I set port to blank it returns 9200.
If I try to use Search
>Search(index="cvr-permanent", type="virksomhed", q='"cvrNummer":"33647093"', size=10)
Error in curl::curl_fetch_memory(url, handle = handle) :
Failed to connect to distribution.virk.dk port 9200: Timed out
(elastic maintainer here)
You should be able to pass in httr::authenticate() to elastic::Search and other functions from the pkg, e.g,.
x <- Search(config = c(httr::verbose(), authenticate("foo", "bar")))
You should see the Authorization: Basic XXXXXX header in the request headers
does that work?
Related
I am attempting to connect to an SFTP site to pull data. It used to work, but for some reason, it stopped working a couple of weeks ago. The owners of the SFTP say nothing has changed on their end, and I can pull data easily without error using WinSCP.
protocol <- "sftp"
server <- "sftp.xxxx.net"
userpwd <- "user:password"
file <- "/public/bpus_dailytx.csv"
url <- paste0(protocol, "://", server, file)
data <- getURL(url = url, userpwd=userpwd, verbose = TRUE)
When I run this, I now get the following info:
* Trying xxx.xx.xx.xxx...
* Connected to sftp.xxxx.net (xxx.xx.xx.xxx) port 22 (#0)
* SSH MD5 fingerprint: 34rh3ie93hhr39hhdik3
* SSH authentication methods available: publickey,keyboard-interactive
* Using SSH public key file '(nil)'
* Using SSH private key file ''
* SSH public key authentication failed: Unable to extract public key from private key file: Unable to open private key file
* No identity would match
* Authentication failure
* Closing connection 0
Error in function (type, msg, asError = TRUE) : Authentication failure
It connects OK but then the authentication fails. Any ideas what could be going on here? Again, this code used to work but something has changed. What are some other ways I can attempt to pull the data other than this?
Edit: WinSCP screenshots:
It is difficult to say why your code stopped working because we do not have enough information about configurations (both on your machine and on the server) when it was working.
Because your keyfile was not in the proper format for RCurl, my leading hypothesis is that although the server folks said nothing changed on their end, I think they removed the password authentication option. That is because your code attempts password authentication only. If password authentication were still available, the one line in your output would look something like this:
SSH authentication methods available: publickey,password,keyboard-interactive
It is, as you noted, now:
SSH authentication methods available: publickey,keyboard-interactive
Therefore, the solution here was to convert your keyfile from PuTTY to OpenSSH format using PuTTYgen and then use the following RCurl code pointing to your new keyfile:
protocol <- "sftp"
server <- "sftp.xxxx.net"
file <- "/public/bpus_dailytx.csv"
url <- paste0(protocol, "://", server, file)
keypasswd <- "your_keypasswd"
ssh.private.keyfile = "your_path_to_keyfile"
username <- "your_username"
data <- getURL(url = url, keypasswd = keypasswd, ssh.private.keyfile = ssh.private.keyfile, username = username, verbose = TRUE)
And I will add a special thanks to #Tensibai for the assistance. It would have taken me way longer to arrive at this solution without their insight with the keyfile format.
I'm writing a R package for the first time. Its called aactr and is hosted on my GitHub: https://github.com/jasonbaik94/aactr
Right now, the package only has one function, aact_connect:
aact_connect <- function(user, password) {
drv <- DBI::dbDriver('PostgreSQL')
con <- DBI::dbConnect(drv,
dbname="aact",
host="aact-db.ctti-clinicaltrials.org",
port=5432,
user=user,
password=password)
}
My concern is that users of my package wouldn't like to input their username and password in their R script for privacy reasons.
What would be a nice workaround to ensure user privacy?
One thought I had: When the user types aact_connect(), a window pops up where the user can input username and password and press Enter, after which the connection would be made. Also, for those without a username or password, I'd put a parameter, init_connection = TRUE, after which this sign up page would load: https://aact.ctti-clinicaltrials.org/users/sign_up
Any others suggestions are greatly welcome!
There are many ways to securely handle database credentials in scripts. See few examples below:
Use configuration files such as yaml saves securely on user's machines for R to read into needed variables. See #Spacedman's answer.
config.yaml
db:
host : localhost
port : 5432
name : mypgdb
user : pg_useR
pwd : ***
R
library(yaml)
config = yaml.load_file("/path/to/config.yml")
dbConnect(drv, host = config$db$host, port = config$db$port,
dbname = config$db$name,
user = config$db$user, password = config$db$pwd)
Use environmental variables that are linked to user or system profiles:
db_creds <- Sys.getenv(c("DB_HOST", "DB_PORT", "DB_NAME", "DB_USER", "DB_PWD"))
con <- DBI::dbConnect(drv,
dbname = db_creds[['DB_NAME']],
host = db_creds[['DB_HOST']],
port = db_creds[['DB_PORT,']],
user = db_creds[['DB_USER']],
password = db_creds[['DB_PWD']])
Use DSNs that maintains connection parameters but requires an ODBC connection which can be run with the generalized odbc package (part of same DBI family as RPostgreSQL). Postgres maintains up-to-date odbc drivers for most operating systems. See R-bloggers post.
odbc.ini
[myPG_DSN]
Driver = PostgreSQL Unicode
Database = mypg_db
Servername = localhost
UserName = pg_useR
Password = ***
R
db <- dbConnect(odbc::odbc(), "myPG_DSN")
I am looking to connect to a server using the mongolite package of R but I have the following error
Error: No suitable servers found (serverSelectionTryOnce set): [connection timeout calling ismaster on 'XXX.XX.XX.XXX:XXXX'] where XX are the IP of the server.
Additionaly I have tried to use a proxy to reach out the required server without success.
I have try to create a .Renvion file in my user/Home
and adding the folowing line into this conf file
options(internet.info = 0)
http_proxy=//proxy.serv:0000/ where 0000 are the port of the proxy
Then I am calling the require serveur with its URL in the command line below
mongo(url = "mongodb://XXX.XX.XX.XXX:XXXX", db = "TEST",collection = "TEST1")
without success.
I have allowed the firewall security to the required serveur going forward.
Any Help?
I am trying to query AWS ElasticSearch Service (AWS ES) through a package in R called elastic. I am getting an error when trying to connect to the server.
Here is an example:
install.packages("elastic")
library(elastic)
aws_endpoint = "<secret>"
# I am certain the endpoint exists and is correct, as it functions with Kibana
aws_port = 80
# I have tried 9200, 9300, and 443 with no success
connect(es_host = aws_endpoint,
es_port = 80,
errors = "complete")
ping()
Search(index = "foobar", size = 1)$hits$hits
Whether pinging the server, or actually trying to search a document, both retrieve this error:
Error: 404 - no such index
ES stack trace:
type: index_not_found_exception
reason: no such index
resource.type: index_or_alias
resource.id: us-east-1.es.amazonaws.com
index: us-east-1.es.amazonaws.com
I have gone into my AWS ES dashboard and made certain I am using indexes that exist. Why this error?
I imagine I am misunderstanding something about transport protocols. elastic interacts with elasticsearch's HTTP API. I thought this was fine.
How do I establish an approriate connection between R and AWS ES?
R version 3.3.0 (2016-05-03); elastic_0.7.8
Solved it.
es_path must be specified as an empty string (""). Otherwise, connect() understands the AWS region (i.e. us-east-1.es.amazonaws.com) as the path. I imagine connect() adds the misunderstood path in the HTTP request, following the format shown here.
connect(es_host = aws_endpoint,
es_port = 80,
errors = "complete",
es_path = ""
)
Just to be clear, the parameters I actually used is shown below, but they should not make a difference. Fixing es_path is the key.
connect(es_host = aws_endpoint,
es_port = 443,
es_path = "",
es_transport_schema = "https")
I am making the move from RSQLite to RMySQL and I am confused by the user and password fields. FWIW, I'm running Windows 7, R 2.12.2, MySQL 5.5 (all 64 bit), and RMySQL 0.7-5.
I installed RMySQL as prescribed in this previous SO question, and as far as I know it works (i.e., I can load the package with library(RMySQL)). But when I try to run the tutorial from the R data import guide, I get a "failed to connect to database..." error. This is the code from the tutorial from the guide:
library(RMySQL) # will load DBI as well
## open a connection to a MySQL database
con <- dbConnect(dbDriver("MySQL"), user = "root", password = "root", dbname = "pookas")
## list the tables in the database
dbListTables(con)
## load a data frame into the database, deleting any existing copy
data(USArrests)
dbWriteTable(con, "arrests", USArrests, overwrite = TRUE)
dbListTables(con)
## get the whole table
dbReadTable(con, "arrests")
## Select from the loaded table
dbGetQuery(con, paste("select row_names, Murder from arrests",
"where Rape > 30 order by Murder"))
dbRemoveTable(con, "arrests")
dbDisconnect(con)
On the second line I get the following error:
> con <- dbConnect(dbDriver("MySQL"), user = "richard", password = "root", dbname = "pookas")
Error in mysqlNewConnection(drv, ...) :
RS-DBI driver: (Failed to connect to database: Error: Access denied for user 'richard'#'localhost' (using password: NO)
)
I have tried with and without user and password and with admin as user. I have also tried using a dbname that I made before with the command line and with one that doesn't exist.
Any tips? Is there a good reference here? Thanks!
That is most likely a setup issue on the server side. Make sure that networked access is enabled.
Also, a local test with the command-line client is not equivalent as that typically uses sockets. The mysql server logs may be helpful.
First try to connect to MySQL server using MySQL Workbench or command line mysql using the same parameter. If it connects then R should also be able to connect.
Typically this issue comes when MySQL server doesn't allow connections from remote machines.
As people have told you, you can try to connect to the host with other application as mysql workbench. How odd! When I have tried in RStudio to connect to my db with your code without indicate the host in the command I haven't been able to connect.
I have needed to indicate the host ( host = 'localhost' ) in the command.