How to connect to PostgreSQL running in Docker container from local R? - r

I have a local Docker container running PostgreSQL. I want to be able to connect to and interact with this database from R running on my host machine (Mac OS).
I can connect using pgadmin4 via the following address
http://0.0.0.0:5434/browser/
then adding a new server:
Add new server. General Tab --> name: tagbase. Connection Tab --> Host name/address: postgres. Connection Tab --> Port: 5432. Connection Tab --> Maintenance database: postgres. Connection Tab --> Username: tagbase
This works perfectly.
However, to connect from R I try:
require("RPostgreSQL")
# load the PostgreSQL driver
drv <- dbDriver("PostgreSQL")
# create a connection to the postgres database
con <- RPostgreSQL::dbConnect(drv, dbname = "postgres",
host = "localhost", port = 5434,
user = "tagbase", password = "tagbase")
This attempt simply hangs until it crashes R.
Perhaps a viable solution is something similar to this. Many thanks for any help.
EDIT - 20190207
Thanks for the comments. I have made the changes with no improvement but agreed the changes were necessary.
I successfully start this docker network (of 3 containers) via terminal as below. It looks to me like I want to connect to the postgres container at 0.0.0.0 on port 5432, correct?
$ docker-compose up
Starting tagbase-server_postgres_1_3f42d4fc1a77 ... done
Starting tagbase-server_pgadmin4_1_52ab92a49f22 ... done
Starting tagbase-server_tagbase_1_9d3a22c8be46 ... done
Attaching to tagbase-server_postgres_1_3f42d4fc1a77, tagbase-server_pgadmin4_1_52ab92a49f22, tagbase-server_tagbase_1_9d3a22c8be46
postgres_1_3f42d4fc1a77 | 2019-02-05 19:35:45.999 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
I thought I was connecting to the server via R exactly as I've done using pgadmin but the following doesn't seem to work:
# create a connection to the postgres database
con <- DBI::dbConnect(RPostgreSQL::PostgreSQL(), dbname = "postgres",
host = "0.0.0.0", port = 5432,
user = "tagbase", password = "tagbase")
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect tagbase#0.0.0.0:5432 on dbname "postgres":
FATAL: role "tagbase" does not exist)
I now realize pgadmin is also running in the docker container network. Thus, local host for the pgadmin connection is the database server. Seems like I need a solution like this
Note the source for the docker builds is here following the instructions here.

If you want to connect directly to a postgres database inside a docker from outside docker world, you must expose a port on postgres docker. So first, you need to edit the file "Dockerfile-postgres", and add EXPOSE 5432
FROM postgres:10
COPY ./sqldb/tagbase-schema.sql /docker-entrypoint-initdb.d/
# Expose default postgres port
EXPOSE 5432
Then build and run the dockers according to the provided instrucctions (Checked on October 6, 2019)
$ docker-compose build
$ docker-compose up
Add the database using pgAdmin
Add New Server
 General Tab --> name: tagbase
 Connection Tab --> Host name/address: postgres
 Connection Tab --> Port: 5432
 Connection Tab --> Maintenance database: postgres
 Connection Tab --> Username: tagbase
Edit your R scritp according to the databse name and port:
# install.packages('RPostgreSQL')
library(RPostgreSQL)
# load the PostgreSQL driver
drv <- dbDriver("PostgreSQL")
# create a connection to the postgres database
con <- RPostgreSQL::dbConnect(drv, dbname = "tagbase",
host = "localhost", port = 5432,
user = "tagbase", password = "tagbase")
# Test query
temp <- dbGetQuery(con, 'select * from public.metadata_types')
# Evaluate output
str(temp)
# 'data.frame': 142 obs. of 8 variables:
# $ attribute_id : num 1 2 3 4 5 6 7 8 9 10 ...
# $ category : chr "instrument" "instrument" "instrument" "instrument" ...
# $ attribute_name: chr "instrument_name" "instrument_type" "firmware" "manufacturer" ...
# $ type : chr "string" "string" "string" "string" ...
# $ description : chr "Append an identifer that is unique within your organization. This is essential if a device is recycled." "Type of instrument" "Version number of the firmware used to build the device" "Name of manufacturer" ...
# $ example : chr "16P0100-Refurb2" "archival, popup, satellite, acoustic tag, or acoustic receiver" NA "Wildlife Computers, Microwave Telemetry, Lotek Wireless, Desert Star Systems, CEFAS, StarOddi, Sea Mammal Resea"| __truncated__ ...
# $ comments : chr "Devices might be reused, so the serial number will be the same. The only way to distinguish is by providing a u"| __truncated__ "Should be restricted to the examples provided." NA NA ...
# $ necessity : chr "required" "required" "required" "required" ...
# Disconnect from database
dbDisconnect(con)

Related

Connecting multiple server PostgreSQL R

I needed to put 3 shards of a database on three different servers. So I created 3 servers in pgAdmin(s1,s2,s3), then I put each server one shard. Then, I tried to connect one of the servers in R; however, I couldn't make the connection. I always get an error:
Error in postgresqlNewConnection(drv, ...) : RS-DBI driver: (could not connect postgres#172.17.0.1:5432 on dbname "postgres": could not connect to server: Operation timed out Is the server running on host "172.17.0.1" and accepting TCP/IP connections on port 5432?
My code is:
#install.packages("RPostgreSQL")
require("RPostgreSQL")
library(DBI)
# create a connection
# save the password that we can "hide" it as best as we can by collapsing it
pw <- {
"postgres"
}
# loads the PostgreSQL driver
drv <- dbDriver("PostgreSQL")
# creates a connection to the postgres database
con <- dbConnect(
drv,
dbname = "postgres",
host = "172.17.0.1",
port = 5432,
user = "postgres",
password = pw
)
rm(pw) # removes the password
pgAdmin snap
Did I write something wrong?
if this is using container make sure to forward the port 5432 on 0.0.0.0 i.e the container is listening on the port 5432.
Also you've gotta check this setting if you are not doing the connection locally ONLY>, in the postgresql.conf file:
# - Connection Settings -
#listen_addresses = 'localhost' >>>> This should be = '*' instead of localhost
Save the conf and restart the service. Hope this helps!

Connecting Rstudio with PostgreSQL in AWS EC2

I am currently running PostgreSQL server in AWS EC2 (Ubuntu) and having trouble accessing it from local server. How can I access remote database from local RStudio?
Code in R:
library(RPostgreSQL)
library(DBI)
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, user='postgres', password='password', dbname='dvd', host='ec2-xx-xxx-xxxx-xxxx.ap-northeast-2.compute.amazonaws.com',port=5432)
dbListTables(con)
Result:
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect postgres#ec2-xx-xxx-xxxx-xxxx.ap-northeast-2.compute.amazonaws.com:5432 on dbname "dvd": FATAL: no pg_hba.conf entry for host "130.17.152.1”, user "postgres", database "dvd", SSL off
)
Calls: <Anonymous> ... eval -> dbConnect -> dbConnect -> postgresqlNewConnection
Execution halted
For AWS security group, I have added an inbound for PostgreSQL with its source set it to anywhere with port range 5432.
By default Postgres does not allow remote connections. To allow remote connections you will need to add an entry to the pg_hba.conf file on the EC2 server. The exact location of pg_hba.conf varies with different installations, but here are some paths to try:
/etc/postgresql/[VERSION]/main/
/var/lib/postgresql/[version]/
Once you've found your pg_hba.conf you need to add a line to allow remote connections, for example.
# TYPE DATABASE USER ADDRESS METHOD
host dvd postgres 130.17.152.1/32 md5
This assumes that your local server IP (130.17.152.1) is static. If not, you could use 0.0.0.0/0 for ADDRESS, also, you can use all for DATABASE and USER like so:
# TYPE DATABASE USER ADDRESS METHOD
host all all 0.0.0.0:0 md5
Depends how tight you want your security to be, but bear in mind this is security at the database level and is completely separate to your AWS security groups.
See https://www.postgresql.org/docs/9.1/auth-pg-hba-conf.html for more information.

PostgreSQL dbConnect to shiny app in ec2 instance

I have an ec2 instance set up with my shiny app and my postgresql database, I want to get the shiny-app to read from the database
If I type psql and \conninfo while ssh-ed into my instance I get
You are connected to database "ubuntu" as user "ubuntu" via socket in "/var/run/postgresql" at port "5432".
When I use R in the ec2 command line and type the following, I can read from my database no problem!
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, dbname = "ubuntu", host = "/var/run/postgresql", port = 5432, user = "ubuntu", password = pw)
However, when I put these same lines in my shiny app.R file I get
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect ubuntu#/var/run/postgresql:5432 on dbname "ubuntu": FATAL: Peer authentication failed for user "ubuntu")
I've tried so many different values for host like
host = "localhost"
host = "my ec2 public ip address"
host = "127.0.0.1"
for example and nothing has been working.
my security group for this ec2 instance has an inboud connection to port 5432.
could this be it: why is one file green and the other pink? the green one is the one that works (local) and the pink one is on my instance
Finally figured it out.. this is the same problem as Getting error: Peer authentication failed for user "postgres", when trying to get pgsql working with rails
except that I was getting a different error for the same underlying problem.
the answer that worked for me is the second one:
1.
nano /etc/postgresql/9.x/main/pg_hba.conf
change peer in this line
local all postgres peer
to
local all postgres trust
Restart the server
sudo service postgresql restart
Login into psql and set your password
psql -U postgres
ALTER USER postgres with password 'your-pass';
Finally change the pg_hba.conf from
local all postgres trust
to
local all postgres md5
and that finally worked

RStudio on Windows PostgreSQL SSL Connection with RPostgreSQL

I am having trouble creating an SSL connection using RPostgreSQL to an AWS hosted PostgreSQL database.
Here is what I've tried so far:
Created the PostgreSQL database on AWS.
Set the database parameter "rds.force_ssl" to 1.
Downloaded the AWS public key from https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
Test the connection from a windows command prompt with psql (it works).
Executed the following in R:
library(RPostgreSQL)
cert <- paste0("C:/Users/johnr/Downloads/", "rds-combined-ca-bundle.pem")
dbname <- paste0("dbname=", "flargnog", " ", "sslrootcert=", cert, " ", "sslmode=verify-full")
host <- "xxxxxx.xxxxx.us-region-2.rds.amazonaws.com"
con <- dbConnect(dbDriver("PostgreSQL"), user="username", host=host, port=5432, dbname=dbname, password="abcd1234!")
I receive an error message after executing the last statement:
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect username#xxxxxx.xxxxx.us-region-2.rds.amazonaws.com on dbname "flargnog"
If I change the rds.force_ssl setting to 0 (and remove the ssl stuff from dbname) the connection works just fine.
I have looked at other posts on Stackoverflow related to this issue. This and this seem to indicate an SSL connection is not possible due to issues with RPostgreSQL. However, this post indicates that you can.
Any guidance would be appreciated!
You can try to ssh to the rds instance using e.g. putty and port-forward your local port 5432 to the remote port 5432. Once the ssh connection is open in R just connect to localhost:5432...
Here is how to port-forward using putty:
http://www.akadia.com/services/ssh_putty.html
Here is how this works via command-line:
https://gist.github.com/magnetikonline/3d239b82265398568f31
P.S.: Make sure your instance is in a security-group that accepts ssh connections - port 22

Importing files from PostgreSQL to R

I have a large data-set and I will preform some analysis in R software.
While I could not import the data properly to R.
I get this error:
Error in postgresqlNewConnection(drv, ...) : RS-DBI driver: (could not connect User#local on dbname "Intel"
I have used PostgreSQL to open data and somehow manage it. How can I import the existing data in the PostgreSQL to the R software?
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, host='localhost', port='5432', dbname='Swiss',
user='postgres', password='123456')
Moreover, "RPostgreSQL" package in R should be installed.
Try the R package RPostgreSQL http://cran.r-project.org/web/packages/RPostgreSQL/ .
You can see how to use it in http://code.google.com/p/rpostgresql/ .
Example:
library(RPostgreSQL)
drv <- dbDriver("PostgreSQL") ## loads the PostgreSQL driver
con <- dbConnect(drv, dbname="R_Project") ## Open a connection
rs <- dbSendQuery(con, "select * from R_Users") ## Submits a statement
fetch(rs,n=-1) ## fetch all elements from the result set
dbGetQuery(con, "select * from R_packages") ## Submit and execute the query
dbDisconnect(con) ## Closes the connection
dbUnloadDriver(drv) # Frees all the resources on the driver
You have to configure two things on the PostgreSQL server before you are able to connect remotely. This is a instruction how to configure this under Linux:
1. Find and configure postgresql.conf to allow the TCP service to accept connections from any host, not only localhost
find / -name "postgresql.conf"
In my linux OS the file is locate in /etc/postgresql/9.6/main/, so I modify it there. Add the line "listen_addresses = '*'" as follows:
/etc/postgresql/9.6/main/postgresql.conf
#listen_addresses = 'localhost' # what IP address(es) to listen on;
# insert the following line
listen_addresses = '*'
2. Find and configure pg_hba.conf to allow to connect with a client from any host
sudo find / -name "pg_hba.conf"
In my linux OS the file is locate in /etc/postgresql/9.6/main/, so I modify it there. Add the line "host all all 0.0.0.0/0" as follows:
sudo nano /etc/postgresql/9.6/main/pg_hba.conf
# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records. In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.
#
# insert the following line
host all all 0.0.0.0/0 trust
3. Stop and start the server
sudo service postgresql stop
sudo service postgresql start
4. Connect with you client, now it should work.
GOOD LUCK!

Resources