I have my password to database stored in pgpass.conf file. I am connecting to database from R with RPostgres, without specifying password so it is read from pgpass.conf, like this:
con <- dbConnect(RPostgres::Postgres(),
dbname = "dbname",
user = "username",
host = "localhost",
port = "5432")
It usually works perfectly, however when I try to connect to database from shiny app it doesn't work. Connection definition is exactly the same as above and placed in server.R script. When I run Shiny app with default arguments I get an error:
FATAL: password authentication failed for user "username"
password retrieved from file "C:\Users\...\AppData\Roaming/postgresql/pgpass.conf"
When password is explicitly given in connection definition:
con <- dbConnect(RPostgres::Postgres(),
dbname = "dbname",
user = "username",
host = "localhost",
password = "mypass",
port = "5432")
everything works.
To make things stranger, when port for shiny is set to some value, for example: shiny::runApp(port = 4000), connection is established without specifying password, but ONLY for the first time - that means when app is closed and reopened in the same R session, the error occurs again.
I've tested package 'RPostgreSQL' - it doesn't work neither, only error message is different:
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect postgres#localhost on dbname "dbname")
I use 32-bit R but I've tested it on 64-bit and it was the same. Shiny app was run both in browser (Chrome) and in Rstudio Viewer.
Here my session info:
R version 3.2.2 (2015-08-14)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
locale:
[1] LC_COLLATE=Polish_Poland.1250 LC_CTYPE=Polish_Poland.1250 LC_MONETARY=Polish_Poland.1250
[4] LC_NUMERIC=C LC_TIME=Polish_Poland.1250
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] RPostgres_0.1 DBI_0.3.1.9008 shiny_0.12.2
loaded via a namespace (and not attached):
[1] R6_2.1.1 htmltools_0.2.6 tools_3.2.2 rstudioapi_0.3.1 Rcpp_0.12.1.3 jsonlite_0.9.17 digest_0.6.8
[8] xtable_1.7-4 httpuv_1.3.3 mime_0.4 RPostgreSQL_0.4
There's likely something different about the environment in which the command is run between Shiny and your system R GUI. I get around this by storing my credentials in an Renviron file:
readRenviron("~/.Renviron")
con <- dbConnect(RPostgres::Postgres(),
dbname = Sys.getenv('pg_db'),
user = Sys.getenv('api_user'),
...)
The thing about that is you could maintain separate Renvirons for staging and production environments. This allows your script to take a commandArgs() to specify which DB credentials it should use:
#!/usr/bin/env Rscript
environ_path <- switch(commandArgs(),
'staging' = {"~/staging.Renviron"},
'production' = {"~/production/Renviron"})
readRenviron(environ_path)
Then from BASH:
Rscript analysis.R staging
The Error is in Postgresql:
C:\Users\...\AppData\Roaming/postgresql/pgpass.conf
the file path contains "/" instead of "\"
Related
I have seen many questions on curl's timeout problem. However, I am still confused about my case.
My problem occurred when I was using quantmod::getSymbols. Every trial ended up with
Warning message:
x = 'AAPL', get = 'stock.prices': Error in curl::curl_fetch_memory(cu, handle = h): Timeout was reached: [finance.yahoo.com] Operation timed out after 10008 milliseconds with 0 out of 0 bytes received
Note I am using a proxy. I have tried to switch on and off the proxy or run
httr::set_config(httr::use_proxy(
"127.0.0.1:xxxxxx", port = 8080,
username = "xxxx", password = "****"
), override = TRUE)
However, nothing works.
After getting confused about quantmod's internal details, I decided to experiment on the pure curl_fetch_memory.
the example in curl_fetch_memory's document works normally on my computer
curl::curl_fetch_memory("http://httpbin.org/cookies/set?foo=123&bar=ftw")
curl_fetch_memory does not work for "https://finance.yahoo.com/" (Note that I cannot get access to the website on a web browser without my proxy)
The followings codes can successfully fetch results from https://finance.yahoo.com/:
curl_opts <- list(
ssl_verifypeer = 0L,
proxyuserpwd = "xxxx:****",
proxyauth = as.integer(15),
proxy = "127.0.0.1:xxxxxx",
proxyport = 8080
)
cookie_handler <- curl::new_handle()
curl::handle_setopt(handle=cookie_handler, .list=curl_opts)
curl::curl_fetch_memory("https://finance.yahoo.com/",
handle = cookie_handler)
It seems that quantmod is unable to use the proxy setting properly. However, there is no option to set the proxy inside the functions from the package. How can I solve my problem?
System:
R version 4.1.1 (2021-08-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19043)
Matrix products: default
locale:
[1] LC_COLLATE=Chinese (Simplified)_China.936
[2] LC_CTYPE=Chinese (Simplified)_China.936
[3] LC_MONETARY=Chinese (Simplified)_China.936
[4] LC_NUMERIC=C
[5] LC_TIME=Chinese (Simplified)_China.936
# ...
I think I have found the solution. All you need is to switch on the proxy and run
Sys.setenv(ALL_PROXY = "YOUR_PROXY_HOST")
before the request.
However, currently I still have no idea of how the curl uses the setting ALL_PROXY. I would appreciate it if anyone can offer it an explanation.
I've looked at https://github.com/ropensci/RSelenium/issues/94 and https://github.com/ropensci/RSelenium/issues/82 but was not able to solve my problem. It didn't help that this person was on Windows, and I am on Mac (El Capitan, version 10.11.6)
I am trying to learn data scraping with RSelenium, but some of the technical aspects of it are giving me issues early on. I have a few questions first and then will share my code:
(1) Right away, it says that startServer() is deprecated. specifically, that:
startServer()
# output
Warning message:
startServer is deprecated.
Users in future can find the function in
file.path(find.package("RSelenium"), "example/serverUtils").
The sourcing/starting of a Selenium Server is a users responsiblity.
Options include manually starting a server see
vignette("RSelenium-basics", package = "RSelenium")
and running a docker container see
vignette("RSelenium-docker", package = "RSelenium")
.
what should i use in place of startSever(), or what do I need to change on my computer? I'm confused as to what this warming message is saying.
(2) Since it's just a warning, I continue by trying to open a browser in chrome. I quickly run into another error:
remDr = remoteDriver$new(browserName = 'chrome')
remDr$open()
# output
[1] "Connecting to remote server"
$webdriver.remote.sessionid
[1] "4d0ad1d9-1c4b-4171-8dce-ba8363f5849e"
$locationContextEnabled
[1] TRUE
$webStorageEnabled
[1] TRUE
$takesScreenshot
[1] TRUE
$javascriptEnabled
[1] TRUE
$message
[1] "session not created exception\nfrom unknown error: Runtime.executionContextCreated has invalid 'context': {\"auxData\":{\"frameId\":\"34144.1\",\"isDefault\":true},\"id\":1,\"name\":\"\",\"origin\":\"://\"}\n (Session info: chrome=54.0.2840.71)\n (Driver info: chromedriver=2.20.353124 (035346203162d32c80f1dce587c8154a1efa0c3b),platform=Mac OS X 10.11.6 x86_64)"
$hasTouchScreen
[1] TRUE
$platform
[1] "ANY"
$cssSelectorsEnabled
[1] TRUE
$id
[1] "4d0ad1d9-1c4b-4171-8dce-ba8363f5849e"
the $message line output mentions that the session was not created. on my desktop, what i see is that chrome opens initially for a split second, and then closes / crashes / doesn't actually open up. I try again for firefox, and get:
remDr = remoteDriver$new(browserName = 'firefox')
remDr$open()
# output
[1] "Connecting to remote server"
Selenium message:The path to the driver executable must be set by the webdriver.gecko.driver system property; for more information, see https://github.com/mozilla/geckodriver. The latest version can be downloaded from https://github.com/mozilla/geckodriver/releases
Error: Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
class: java.lang.IllegalStateException
Further Details: run errorDetails method
it is frustrating to try to learn this, but to not even be able to get past the very first steps of opening a browser. Any help is greatly appreciated!
As noted checkForServer and startServer are deprecated you may be able to use them as follows:
unlink(file.path(find.package("RSelenium"), "bin"), recursive = TRUE, force = TRUE)
RSelenium::checkForServer()
For Firefox:
In terminal, run the following command
brew install geckodriver
Running selenium at the default port on Mac has an issue as often Kerberos is already running on default port 4444 on MAC. Run the following command in R console
selServ <- RSelenium::startServer(args = c("-port 5556"))
remDr <- RSelenium::remoteDriver(extraCapabilities = list(marionette = TRUE), port=5556)
remDr$open()
......
# when finished
selServ$stop()
For chrome:
brew install chromedriver
Running selenium at the default port on Mac has an issue. Run the following command in R console
selServ <- RSelenium::startServer(args = c("-port 5556"))
remDr <- RSelenium::remoteDriver(browserName = "chrome",
extraCapabilities = list(marionette = TRUE),
port=5556)
remDr$open()
......
# when finished
selServ$stop()
If the above doesnt help then look at running a Docker container see
http://rpubs.com/johndharrison/RSelenium-Docker and https://github.com/SeleniumHQ/docker-selenium . This basically involves running a Docker container using something like:
$ docker run -d -p 5556:4444 selenium/standalone-chrome:3.0.1-aluminum
then a selenium server and chrome browser should be accessible on port 5556 which you can connect to giving appropriate arguments in remoteDriver.
[Original Post]
I have seen a couple of similar SO questions but I don't think any of them are related to OS.
Long story short, this is what I see when I try to establish a ssl connection to Redshift.
library(RPostgreSQL)
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv,
host = "XXXXXXXX.us-east-1.redshift.amazonaws.com",
port = 5439,
user = "XXXXXXXX",
password = "XXXXXXXX",
dbname = "db1")
> Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect XXXXXXXX#XXXXXXXX.us-east-1.redshift.amazonaws.com on dbname "db1")
I have tried the following and none of them worked.
Connect to Redshift via SSL using R
Importing files from PostgreSQL to R
Connect to Postgres via SSL using R
https://groups.google.com/forum/#!topic/rpostgresql-dev/ELnVUJqjDbk
I have an EC2 node (Ubuntu) in AWS and another Mac Pro in office, so I tried to use the same code to connect from both.
The Linux system works with copying & pasting the exact same code. Here is the Sys.info() on the Linux server:
sysname Linux
release 3.13.0-48-generic
version #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015
nodename ip-xx-xxx-xx-xx
machine x86_64
login unknown
user bcui
effective_user bcui
However, the Mac Pro failed with the exact same error message. Here is the Mac Sys.info() (same as my Mac laptop):
sysname Darwin
release 15.3.0
version Darwin Kernel Version 15.3.0:
Thu Dec 10 18:40:58 PST 2015;
root:xnu-3248.30.4~1/RELEASE_X86_64
nodename bcui-MBP.local
machine x86_64
login bcui
user bcui
effective_user bcui
I am wondering what could be different between the Mac and Linux configuration that causes this error message?
[Update 2016/02/10]
I have tried to uninstall R and all related files, and then reinstall R from homebrew:
brew tap homebrew/science
brew install R
Same error message:
> Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect XXXXXXXX#XXXXXXXX.us-east-1.redshift.amazonaws.com on dbname "db1")
FYI, I can connect to other non-ssl Redshift clusters using the same code, so it could be something related to openssl on Mac.
[Update 2016/02/11]
More info from #MasashiMiyazaki's comment:
I have used psql via CLI to connect and it works fine. In addition, I can also connect successfully using the Python module psycopg2. However, I will have to disable this line from .bash_profile in order for them to work:
export DYLD_LIBRARY_PATH=/usr/lib:$DYLD_LIBRARY_PATH
Taking a shot in the dark here. Have you tried RPostgres?
Install it like so
# install.packages("devtools")
devtools::install_github("RcppCore/Rcpp")
devtools::install_github("rstats-db/DBI")
devtools::install_github("rstats-db/RPostgres")
Then test it out
library(DBI)
library(RPostgres)
con <- dbConnect(RPostgres::Postgres(),
host = "XXXXXXXX.us-east-1.redshift.amazonaws.com",
port = 5439,
user = "XXXXXXXX",
password = "XXXXXXXX",
dbname = "db1")
I've successfully run h2o from R on a linux machine and wanted to install it in Windows too. h2o will not initialise for me. The full output is pasted below but the key seems to be the line
[1] "Failed to connect to 127.0.0.1 port 54321: Connection refused"
curl: (1) Protocol "'http" not supported or disabled in libcurl
Judging from this and this experience it might be something to do with single quotes v double quotes somewhere; but this seems unlikely because then no-one would be able to get h2o / R / Windows combination working and I gather that some people are. On the other hand, this question seems to suggest the problem will be that my curl installation may not have ssl enabled. So I downloaded curl from scratch from this wizard as recommended on the h2o page, selecting the 64 bit version, generic, and selected the version with both SSL and SSH enabled; downloaded it and added the folder it ended up in to my Windows PATH. But no difference.
I've just noticed my Java runtime environment is old and will update that as well. But on the face of it it's not obvious that that could be the problem.
Any suggestions welcomed.
> library(h2o)
> h2o.init()
H2O is not running yet, starting it now...
Note: In case of errors look at the following log files:
C:\Users\PETERE~1\AppData\Local\Temp\Rtmpa6G3WA/h2o_Peter_Ellis_started_from_r.out
C:\Users\PETERE~1\AppData\Local\Temp\Rtmpa6G3WA/h2o_Peter_Ellis_started_from_r.err
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
............................................................
ERROR: Unknown argument (Ellis_cns773)
Usage: java [-Xmx<size>] -jar h2o.jar [options]
(Note that every option has a default and is optional.)
-h | -help
Print this help.
-version
Print version info and exit.
-name <h2oCloudName>
Cloud name used for discovery of other nodes.
Nodes with the same cloud name will form an H2O cloud
(also known as an H2O cluster).
-flatfile <flatFileName>
Configuration file explicitly listing H2O cloud node members.
-ip <ipAddressOfNode>
IP address of this node.
-port <port>
Port number for this node (note: port+1 is also used).
(The default port is 0.)
-network <IPv4network1Specification>[,<IPv4network2Specification> ...]
The IP address discovery code will bind to the first interface
that matches one of the networks in the comma-separated list.
Use instead of -ip when a broad range of addresses is legal.
(Example network specification: '10.1.2.0/24' allows 256 legal
possibilities.)
-ice_root <fileSystemPath>
The directory where H2O spills temporary data to disk.
-log_dir <fileSystemPath>
The directory where H2O writes logs to disk.
(This usually has a good default that you need not change.)
-log_level <TRACE,DEBUG,INFO,WARN,ERRR,FATAL>
Write messages at this logging level, or above. Default is INFO.
-flow_dir <server side directory or HDFS directory>
The directory where H2O stores saved flows.
(The default is 'C:\Users\Peter Ellis\h2oflows'.)
-nthreads <#threads>
Maximum number of threads in the low priority batch-work queue.
(The default is 99.)
-client
Launch H2O node in client mode.
Cloud formation behavior:
New H2O nodes join together to form a cloud at startup time.
Once a cloud is given work to perform, it locks out new members
from joining.
Examples:
Start an H2O node with 4GB of memory and a default cloud name:
$ java -Xmx4g -jar h2o.jar
Start an H2O node with 6GB of memory and a specify the cloud name:
$ java -Xmx6g -jar h2o.jar -name MyCloud
Start an H2O cloud with three 2GB nodes and a default cloud name:
$ java -Xmx2g -jar h2o.jar &
$ java -Xmx2g -jar h2o.jar &
$ java -Xmx2g -jar h2o.jar &
[1] "127.0.0.1"
[1] 54321
[1] TRUE
[1] -1
[1] "Failed to connect to 127.0.0.1 port 54321: Connection refused"
curl: (1) Protocol "'http" not supported or disabled in libcurl
[1] 1
Error in h2o.init() : H2O failed to start, stopping execution.
In addition: Warning message:
running command 'curl 'http://localhost:54321'' had status 1
> sessionInfo()
R version 3.2.3 (2015-12-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
locale:
[1] LC_COLLATE=English_New Zealand.1252 LC_CTYPE=English_New Zealand.1252 LC_MONETARY=English_New Zealand.1252
[4] LC_NUMERIC=C LC_TIME=English_New Zealand.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] h2o_3.6.0.8 statmod_1.4.22
loaded via a namespace (and not attached):
[1] tools_3.2.3 RCurl_1.95-4.7 jsonlite_0.9.19 bitops_1.0-6
We pushed a fix to master for this issue: https://0xdata.atlassian.net/browse/PUBDEV-2526 If you want to try it out now you can build from master as follows:
git clone https://github.com/h2oai/h2o-3
cd h2o-3
./gradlew build -x test
R CMD INSTALL ./h2o-r/R/src/contrib/h2o_3.7.0.99999.tar.gz
Or download the next nightly release tomorrow.
I'm trying to use the listBuckets function from the RAmazonS3 package but getting HTTP/1.1 403 Forbidden.
First I'm setting the authentication options as described in the manual:
options(AmazonS3 = c('login' = 'secret'))
I replaced login with my access key ID (20 characters), and secret with my secret access key (40 characters). When I run listBuckets(), I get the following error:
Error in UseMethod("xmlSApply") :
no applicable method for 'xmlSApply' applied to an object of class "NULL"
It's not returning any data, so it must not be connecting properly. Digging into it, there is a getURL call within listBuckets. The output of that line is:
* About to connect() to proxy proxyname.domain.com port xx (#0)
* Trying xxx.xxx.xxx.xxx... * connected
* Connected to proxyname.domain.com (xxx.xxx.xxx.xxx) port xx (#0)
> GET http://s3.amazonaws.com HTTP/1.1
Host: s3.amazonaws.com
Accept: */*
Proxy-Connection: Keep-Alive
Date: Fri, 12 Sep 2014 09:02:41 EDT
Authorization: AWS [login]:[unknown 27-character code]=
< HTTP/1.1 403 Forbidden
< x-amz-request-id: [unknown 16-character code]
< x-amz-id-2: [unknown 64-character code]
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Fri, 12 Sep 2014 13:02:40 GMT
< Server: AmazonS3
< Cache-Control: proxy-revalidate
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
<
* Connection #0 to host proxyname.domain.com left intact
Any ideas where I'm going wrong?
I'm not sure what value to use for login, so I've also tried my AWS account name, my AWS account user name, and literally 'login', but get a similar error.
Before finding the RAmazonS3 package, I was starting to write my own S3api functions using the AWS Command Line Interface. I was able to successfully list the buckets and their objects when calling the AWS CLI from R.
sessionInfo()
R version 3.1.1 (2014-07-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] RCurl_1.95-4.3 bitops_1.0-6 RAmazonS3_0.1-5
loaded via a namespace (and not attached):
[1] digest_0.6.4 tools_3.1.1 XML_3.98-1.1
Just to repeat what's in the comment above; i have noticed that most of the R-packages that connected to AWS service were out of date. So i have created a new package AWSConnect that allows a user to do most basic operations with S3 and EC2. In that package, the function s3.ls() is designed to list the bucket on S3.
Please feel free to use it, and report any bugs/request/issues
The package works in most cases:
login = AWS secret key
secret = AWS secret access key
Example:
markus_test is a new created bucket with no public permission
auth <- c("AKIAJN6VFFXXXXXXXXXX" ="d95ij4uy0i6n+auvhwLLP6VQiz27OdXXXXXXXXXX")
listBucket("markus_test", auth)`
Key LastModified ETag Size
1 rmr2_example.R 2014-09-15 23:38:48 c7f4544cf972bed52fa84164cf2505bf 1248
Owner.ID Owner.DisplayName
1 2a22982b6e7216f42abd2e8848f07a8ada0b1c11318dc8331aee068f29b7765d markusataws
StorageClass
1 STANDARD
> sessionInfo()
R version 3.1.1 (2014-07-10)
Platform: x86_64-apple-darwin10.8.0 (64-bit)
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] RAmazonS3_0.1-5
loaded via a namespace (and not attached):
[1] digest_0.6.4 RCurl_1.95-4.3 tools_3.1.1 XML_3.98-1