I am using R on Windows to connect to a PostgreSQL database hosted on AWS. The database is set up using forcessl = 1 - this means that any connection needs to be set up with sslmode=require.
The base RPostgreSQL package does not provide any exposure to ssl options. This has been raised as an issue many times (see here, here, here and here)
I know there is a workaround using the RPostgres package, but for other functionality reasons I would much prefer to use the RPostgreSQL package.
A few answers (e.g. here) have proposed using a modified dbname to connect with ssl like so:
dbConnect(dbDriver('PostgreSQL'),
dbname = 'dbname=foobar sslmode=require', # modified dbname
host = 'foobar.rds.amazonaws.com',
port = 5439,
user = 'foobar',
password = 'foobar')
But this did not work for me using the CRAN version of the package. This led me to a recent issue raised on the RPostgreSQL github: https://github.com/tomoakin/RPostgreSQL/issues/88
The initial user was able to use the modified dbname method when he compiled the package from source. On Windows, using the latest source package (0.6.2) compiled with RTools, I get the following error when I run the modified dbname code:
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect xxxxx.rds.amazonawss.com:5432 on dbname "xxxxxxx": sslmode value "require" invalid when SSL support is not compiled in
)
From this and the rest of the thread, it looks like SSL is not possible from current source in both Windows and Mac. However, the developer suggests:
If you compile in a environment where libssl and libpq was made SSL activated form, then the driver can use SSL.
I think this means I could manually download the libs and compile myself, but I am not sure if it is a quick fix or if it would require significant rewriting of the package. Any help or pointing in the right direction would be much appreciated. How can I do this in a safe, repeatable way?
I was able to solve this for the El Capitan macOS R users in my office, by doing the following:
Remove RPostgreSQL R package if you already have installed. Methods vary on how to do this, but from either R.app console or R in Terminal, type remove.packages('RPostgreSQL')
Make sure you have Homebrew installed, and from Terminal run: brew install libpq openssl
Open R.app, and from the Packages & Data menu, select Package Installer.
From the first drop-down menu, choose CRAN (sources) (choose mirror closest to you if you haven't used this before).
Using package search, find RPostgreSQL and for the options below, keep At System Level checked, and check Install Dependencies, then click Install Selected.
Quit out of all R and RStudio programs, and try using the new from source installed RPostgreSQL package.
DISCLAIMER: If you have heavy compile dependencies on OpenSSL or libpq for other programs, I have no idea how doing the above may break other programs.
Building on Windows is a can of worms. See R-Admin Windows Toolset. The only openSSL binaries for windows are from unknown developers. Building and installing openSSL on windows is another can of worms that you will need to research. It might be easier to install openSSL inside the R Windows build environment, but I have no experience with that.
EDIT: It turns out that when installing postgres on Windows, postgres installs openSSL. That means that the central problem on Windows is installing the Windows Toolset for R, installing postgres, then pointing the R build system to libpq.
Another solution would be to run linux in a virtual machine under windows. Here is one way to Install linux on Windows. With linux, depending on the distribution, you would only need to do something like the following from the linux command line (for a RedHat variant of linux):
sudo yum install openSSL
sudo yum install postgresql96
sudo yum install R
Line 2 installs libpq which is required for RPostgreSQL. It is libpq which must be compiled with openSSL. You will only be installing and using the PostgreSQL client, not the server and will also get psql. There might be other packages required, see R linux toolset. Normally, these will get pulled in with the above and should not be a problem.
RPostgreSQL contains a version of libpq, but the compile script does not look like it checks for openSSL, at least not on macOS. So it is important to get a system provided libpq installed.
It is also important for the RPostgreSQL configure script to find pg_config, which is installed when postgres client is installed. Not sure about windows through. So make sure pg_config is in your path. Type pg_config to find out.
Now you need to download and compile RPostgreSQL. To start R, type the following at the linux terminal.
R
Then from within R, get, compile and install RPostgreSQL:
install.packages("https://cran.r-project.org/src/contrib/RPostgreSQL_0.6-2.tar.gz", repo=NULL, type="source")
This should compile and install this version of RPostgreSQL. This last line should also work in windows if you have the windows tools installed correctly.
Hopefully, this gives you some ideas.
Related
I'm having issues getting the ODBC driver for Snowflake to work on an M1 Apple Silicon Mac running Big Sur.
Successfully following the instructions on Snowflake's website gets me to the point where testing the driver from the command line (using iodbctest) using the DSN results in the following error:
1: SQLDriverConnect = [iODBC][Driver Manager]dlopen(/opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib, 6): no suitable image found. Did find:
/opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib: no matching architecture in universal wrapper
/opt/snowfl (0) SQLSTATE=00000
2: SQLDriverConnect = [iODBC][Driver Manager]Specified driver could not be loaded (0) SQLSTATE=IM003
My Snowflake driver is installed to /opt/snowflake/snowflakeodbc, so that is correct -- I'm suspicious that this is specifically an M1 problem. I'm using the 2.24.1 version of the driver available from the download mirror here, and the path to the driver in /etc/odbcinst.ini is /opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib (which exists and seems, from all my research, that it should be right).
When I run a connection via DBI in R, I get a completely different error:
Error: nanodbc/nanodbc.cpp:1021: 00000:
[Snowflake][ODBC] (11560) Unable to locate SQLGetPrivateProfileString function.
In other StackOverflow posts, people have referenced the above error meaning that there is a missing library of some kind (IODBC isn't configured correctly?), but I've tried quite a few things to no avail. Any guidance would be great.
Tinkered with this a bit more and realized it's an artifact of the installation pathways for the .dmgs & the preset paths in simba.snowflake.ini.
You need to point the Snowflake driver towards the iODBC dylib (as per a sideswiping statement in the docs) -- the driver is originally configured to look for the ODBC dylib (not iODBC) in a folder that's on the path.
When you install the iODBC driver, verify that it is installed to /usr/local/iODBC (this was where my Silicon Mac installed it to) -- and that /usr/local/iODBC/lib has libiodbc.dylib in it. If so, navigate to your installed snowflake driver directory (should be /etc/snowflake) and alter the simba.snowflake.ini file (/etc/snowflake/snowflake/snowflakeodbc/universal/simba.snowflake.ini). You want to uncomment & alter the last line to be both uncommented & point with a full path towards the iODBC dylib (instead of the default, which is the ODBC dylib).
# Darwin specific ODBCInstLib
# iODBC
ODBCInstLib=/usr/local/iODBC/lib/libiodbcinst.dylib
Make sure to comment out any other ODBCInstLib line so that only one is configured. That should enable you to get your connection to snowflake up and running on an M1 Mac.
Big Sur is macOS v11.n
Snowflake supports macOS 10.14 and 10.15 Supported OSs
So what you are trying to do is not supported and is unlikely to work
None of the other solutions worked for me but #kiran-kumawat 's answer set me down a path that worked.
It seems like the core of the issue is that the odbc code is looking for arm64 architecture drivers but Snowflake is providing it in x86_64 architecture. By installing an x86_64 versions of odbc we are able to have it successfully talk to the driver.
First I uninstalled R and Rstudio. (it may be possible to sim-link or change things behind the scenes to make this work with existing installs but I am not sure).
Then install rosetta (apples software for translating between architectures) and a version of homebrew built with it. I am leaving my main version of homebrew in place.
softwareupdate --install-rosetta
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
Then use that version of homebrew to install odbc, R, and Rstudio.
arch -x86_64 /usr/local/Homebrew/bin/brew install unixodbc
arch -x86_64 /usr/local/Homebrew/bin/brew install --cask rstudio
arch -x86_64 /usr/local/Homebrew/bin/brew install --cask r
We then need to install the snowflake driver: https://sfc-repo.snowflakecomputing.com/odbc/mac64/latest/index.html
Click through all the install prompts.
Modify your files
/usr/local/etc/odbcinst.ini:
[Snowflake Driver]
Driver = /opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib
/usr/local/etc/odbc.ini
[Snowflake]
Driver = Snowflake Driver
uid = <uid>
server = <server>
role = <role>
warehouse = <warehouse>
authenticator = externalbrowser
We also need to modify the simba.snowflake.ini file.
It is somewhat locked down so run:
sudo chmod 646 /opt/snowflake/snowflakeodbc/lib/universal/simba.snowflake.ini
Then
vim /opt/snowflake/snowflakeodbc/lib/universal/simba.snowflake.ini
And find the ODBCInstLib line that is uncommented and change it to:
ODBCInstLib=/usr/local/Cellar/unixodbc/2.3.9_1/lib/libodbcinst.dylib
After setting this up I was able to use this connection successfully:
install.packages("DBI")
install.packages("odbc")
con <- DBI::dbConnect(odbc::odbc(), "Snowflake")
one of our team member suggested below steps and it worked for us for Apple M1 series
Install the latest snowflake driver
Uninstall m1 based homebrew using cmd
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall.sh)"
Install intel based homebrew - restart terminal when done
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
Re-install unixodbc
arch -x86_64 brew install unixodbc
Test
isql -v Pattern
in your database.yml file for connection to snowflake make following change-
replace "dsn: <DSN_NAME>" with following
conn_str: "Driver={PATH};Locale=en-US;uid={USER_NAME};pwd= {PASSWORD};server=<yours>.snowflakecomputing.com;role=<ROLE>;charset=UTF-8;warehouse=<WAREHOUSE>;database=<DATABASE>;schema=<SCHEMA>;"
Has anyone gotten this to work? I use excel w odbc to refresh snowflake files and have tried multiple ways to move the drivers etc and followed snowflake instructions but never works. I did get parallels to work running windows arm but would prefer to just do this in Mac OS
I also have a M1 (version Monterey 12.0) and I ran into similar issues when I tested the driver. Nevertheless, when I tried the "real connection" it worked like a charm. So, maybe it would be good for you to go and test the "real connection" to avoid a wasting of time using such testing. Hope you find this useful.
I've written an R package that I keep in a (bare) git repository on an SSH server, and I want to be able to install it on a local machine without needing to clone the repository by hand.
I've attempted the following:
devtools::install_git("ssh://user_name#remote/path/to/repository")
but I get the error
Downloading git repo ssh://user_name#remote/path/to/repository
Error in git2r::clone(x$url, bundle, progress = FALSE) :
Error in 'git2r_clone': Failed to start SSH session: Unable to exchange encryption keys
I'm on a Windows 7 machine, using R 3.1.2, git2r version 0.11.0 and devtools version 1.9.1. Any help would be greatly appreciated. Thanks!
Ran into this issue myself. I know this question is a bit old, but for anybody else running into the same issue (like me), here's what I've found.
The Problem
Likely you don't have the library that git2r (the package that devtools uses to interact with git) uses to communicate over SSH.
The Solution
Install it. The example below assumes Ubuntu.
sudo apt-get install libssh2-1 libssh2-1-dev
git2r uses a library called LibSSH2 to enable transport over SSH. You can install it using your package manager if you're on Linux. NOTE if you're on Windows, git2r does not support the SSH protocol yet :/ As of git2r version 0.11.0 (which uses an updated version of libgit2) it looks like SSH is supported on Windows as well. As of this edit, the newest version of git2r is 0.15, so if you don't have SSH support on Windows, try updating git2r (shout out to zeehio for the news).
After you've installed LibSSH2 you'll need to reinstall the git2r package to enable SSH transport (since it's enabled/disabled during the package build).
Sources
Issues on GitHub:
https://github.com/hadley/devtools/issues/796
https://github.com/ropensci/git2r/issues/140
Why do I need LibSSH2 at all? git doesn't use it, right?
You're right! git doesn't use it (to my knowledge). However, libgit2, which is the pure C git API implementation that git2r uses, does.
i am trying to install RODBC with the commadn R CMD INSTALL -l /my/local/path RODBC.tar.gz and it wont find sql.h and sqlext.h ..how do i pass the include and lib paths to this command?
I had a similar problem recently while trying to install RODBC on an instance of Centos 5.8 x64. (Instead of using R CMD install, I just did sudo R, then installed the package inside R - but I was still having the same problem).
I resolved this by installing the following packages using yum:
unixODBC
unixODBC-libs
unixODBC-devel
mysql-connector-odbc
freetds
freetds-devel
Those last two are necessary because I was interfacing with an MSSQL server, which requires TDS. Anyway, once I installed unixODBC and the libs and development package, I was able to install RODBC with no difficulty (again, using sudo R and then install.packages("RODBC").
The only other thing to do is install the correct drivers/libraries for ODBC, which is what mysql-connector-odbc is. You will need to configure unixODBC to meet your needs, but the documentation is pretty solid, so I don't think you'll have too much difficulty.
I'm convinced that using Dirk's package is the best way to install and maintain R on an Ubuntu system. But I want to have some fun and get used to installing R from source.
What are the most common configure flags to use when installing?
Also, if I want to install 2.14.1 and I have 2.14.0 currently installed (which was installed from source), should I first uninstall 2.14.0?
There was a recent thread somewhere about having several versions---one from the apt-get repo, one in /usr/local. Try to find that...
Otherwise, I will roll up 2.14.1 on Friday morning, Michael will do his magic and the repo will have .deb packages of 2.14.1 'real soon', sometimes within a day.
Lastly, you can see which flags are used by getting the package sources for which you just do apt-get source r-base (and that works for any Debian/Ubuntu package that way if you have source references in apt's file.
Edit: By the way, regarding the '64-bit' aspect of your question: Nada. We don't do anything differently. It is "merely" the host OS being more generous with resources. But R finds all it needs to know on its own via its configure etc logic.
I am developing a framework for reproducible computing with R. One problem that I am struggling with is that some R code might run perfectly in version X.Y-Z of a package, but then why you try to reproduce it 3 years later, the packages have updated, some functions are changed, and the code doesn't run anymore. This problem affects also for example Sweave documents that use packages.
The only way to confidently reproduce the results is by installing the R version and version of the packages that were used by the original author. If this was a single case, one could pull stuff from the CRAN archives and install appropriate versions. But for my framework this is impractical, and I need to have the package versions preinstalled.
Assume for now that I restrict myself to a single version of R, e.g. 2.14. What would be a practical way to install many versions of R packages, so that I can load them on the fly? I suppose I can do something like creating separate library directories for every version of every package and then using custom lib.loc arguments while loading them. This is going to be messy though. Any tips or previous attempts to do something similar?
My framework runs on Ubuntu server.
You could install packages with versions (e.g. rename to foo_1.0 directory instead of foo) and softlink the versions you want to re-create a given R + packages snapshot into one library. Obviously, the packages could actually live in a separate tree, so you could have library.projectX/foo -> library.all/foo/1.0.
The operating system gives you even more handles for complete separation, and the Debian / Ubuntu stack as a ton of those available. Two I have played with are
chroot environments: We use this to complete separate build environments from host machines. For example, all Debian uploads I produced are built in a i386 pbuilder chroot hosted on my amd64 Ubuntu server. Chroot is a very powerful Unix system call. Chroots, and particularly the pbuilder system built on top of it (for Debian package building) are meant to operate headless.
Virtual machines: This gives you full generality. My not-so-powerful box easily handles three virtual machines: Debian i386, Ubuntu i386 as well as Windoze XP. For this, I currently use KVM along with libvirt; this is Linux specific. I have also used VirtualBox and VMware in the past.
I would try to modify the DESCRIPTION file, and change the field "Package" there by adding the version number.
For example, you download the package source a from CRAN page (http://cran.r-project.org/web/packages/pls/). Unpack the compressed file (pls_2.3-0.zip) to a directory ("pls/"). The following steps are to change the package name in DESCRIPTION ("pls/DESCRIPTION") and installation with R command 'R CMD INSTALL pls/', where 'pls/' is a path to the package source with modified DESCRIPTION file.
Playing with R library paths seems a dangerous thing to me.