How do I install the ODBC driver for Snowflake successfully on an M1 Apple Silicon Mac? - r
I'm having issues getting the ODBC driver for Snowflake to work on an M1 Apple Silicon Mac running Big Sur.
Successfully following the instructions on Snowflake's website gets me to the point where testing the driver from the command line (using iodbctest) using the DSN results in the following error:
1: SQLDriverConnect = [iODBC][Driver Manager]dlopen(/opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib, 6): no suitable image found. Did find:
/opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib: no matching architecture in universal wrapper
/opt/snowfl (0) SQLSTATE=00000
2: SQLDriverConnect = [iODBC][Driver Manager]Specified driver could not be loaded (0) SQLSTATE=IM003
My Snowflake driver is installed to /opt/snowflake/snowflakeodbc, so that is correct -- I'm suspicious that this is specifically an M1 problem. I'm using the 2.24.1 version of the driver available from the download mirror here, and the path to the driver in /etc/odbcinst.ini is /opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib (which exists and seems, from all my research, that it should be right).
When I run a connection via DBI in R, I get a completely different error:
Error: nanodbc/nanodbc.cpp:1021: 00000:
[Snowflake][ODBC] (11560) Unable to locate SQLGetPrivateProfileString function.
In other StackOverflow posts, people have referenced the above error meaning that there is a missing library of some kind (IODBC isn't configured correctly?), but I've tried quite a few things to no avail. Any guidance would be great.
Tinkered with this a bit more and realized it's an artifact of the installation pathways for the .dmgs & the preset paths in simba.snowflake.ini.
You need to point the Snowflake driver towards the iODBC dylib (as per a sideswiping statement in the docs) -- the driver is originally configured to look for the ODBC dylib (not iODBC) in a folder that's on the path.
When you install the iODBC driver, verify that it is installed to /usr/local/iODBC (this was where my Silicon Mac installed it to) -- and that /usr/local/iODBC/lib has libiodbc.dylib in it. If so, navigate to your installed snowflake driver directory (should be /etc/snowflake) and alter the simba.snowflake.ini file (/etc/snowflake/snowflake/snowflakeodbc/universal/simba.snowflake.ini). You want to uncomment & alter the last line to be both uncommented & point with a full path towards the iODBC dylib (instead of the default, which is the ODBC dylib).
# Darwin specific ODBCInstLib
# iODBC
ODBCInstLib=/usr/local/iODBC/lib/libiodbcinst.dylib
Make sure to comment out any other ODBCInstLib line so that only one is configured. That should enable you to get your connection to snowflake up and running on an M1 Mac.
Big Sur is macOS v11.n
Snowflake supports macOS 10.14 and 10.15 Supported OSs
So what you are trying to do is not supported and is unlikely to work
None of the other solutions worked for me but #kiran-kumawat 's answer set me down a path that worked.
It seems like the core of the issue is that the odbc code is looking for arm64 architecture drivers but Snowflake is providing it in x86_64 architecture. By installing an x86_64 versions of odbc we are able to have it successfully talk to the driver.
First I uninstalled R and Rstudio. (it may be possible to sim-link or change things behind the scenes to make this work with existing installs but I am not sure).
Then install rosetta (apples software for translating between architectures) and a version of homebrew built with it. I am leaving my main version of homebrew in place.
softwareupdate --install-rosetta
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
Then use that version of homebrew to install odbc, R, and Rstudio.
arch -x86_64 /usr/local/Homebrew/bin/brew install unixodbc
arch -x86_64 /usr/local/Homebrew/bin/brew install --cask rstudio
arch -x86_64 /usr/local/Homebrew/bin/brew install --cask r
We then need to install the snowflake driver: https://sfc-repo.snowflakecomputing.com/odbc/mac64/latest/index.html
Click through all the install prompts.
Modify your files
/usr/local/etc/odbcinst.ini:
[Snowflake Driver]
Driver = /opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib
/usr/local/etc/odbc.ini
[Snowflake]
Driver = Snowflake Driver
uid = <uid>
server = <server>
role = <role>
warehouse = <warehouse>
authenticator = externalbrowser
We also need to modify the simba.snowflake.ini file.
It is somewhat locked down so run:
sudo chmod 646 /opt/snowflake/snowflakeodbc/lib/universal/simba.snowflake.ini
Then
vim /opt/snowflake/snowflakeodbc/lib/universal/simba.snowflake.ini
And find the ODBCInstLib line that is uncommented and change it to:
ODBCInstLib=/usr/local/Cellar/unixodbc/2.3.9_1/lib/libodbcinst.dylib
After setting this up I was able to use this connection successfully:
install.packages("DBI")
install.packages("odbc")
con <- DBI::dbConnect(odbc::odbc(), "Snowflake")
one of our team member suggested below steps and it worked for us for Apple M1 series
Install the latest snowflake driver
Uninstall m1 based homebrew using cmd
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall.sh)"
Install intel based homebrew - restart terminal when done
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
Re-install unixodbc
arch -x86_64 brew install unixodbc
Test
isql -v Pattern
in your database.yml file for connection to snowflake make following change-
replace "dsn: <DSN_NAME>" with following
conn_str: "Driver={PATH};Locale=en-US;uid={USER_NAME};pwd= {PASSWORD};server=<yours>.snowflakecomputing.com;role=<ROLE>;charset=UTF-8;warehouse=<WAREHOUSE>;database=<DATABASE>;schema=<SCHEMA>;"
Has anyone gotten this to work? I use excel w odbc to refresh snowflake files and have tried multiple ways to move the drivers etc and followed snowflake instructions but never works. I did get parallels to work running windows arm but would prefer to just do this in Mac OS
I also have a M1 (version Monterey 12.0) and I ran into similar issues when I tested the driver. Nevertheless, when I tried the "real connection" it worked like a charm. So, maybe it would be good for you to go and test the "real connection" to avoid a wasting of time using such testing. Hope you find this useful.
Related
Python: Camelot-PY (Error: GhostscriptNotFound while it is installed) for M1 MacOS
Problems using Camelot-py for PDF parsing. Cannot locate 'gs' package, although it has been installed in any imaginable way... Using M1 Monterey OS. While GS is clearly installed (gs launches from Terminal root just fine), home-brew installation of python / Xcode, PATH variables added through 'export PATH'. Still getting the same error with Ghostscript not being installed when trying to parse a PDF table. Any suggestions / experience on this please? One would have thought it should not be such a massive pain to resolve...
I had the same issue, which was resolved by installing camelot-py from Anaconda: conda install -c conda-forge camelot-py
JFrog not showing any 'Debian Info' for package built on Ubuntu Jammy
I am trying to build a custom debian package of the Cassandra C++ Driver on Jammy (22.04) [0]. It builds fine, but when I manually 'Deploy' the packages to our private JFrog repo no information is shown in the 'Debian Info' section, and the packages are not picked up when I do sudo apt update locally. Locally doing a dpkg-deb -I <package.deb> shows all the expected information. Also, installing the local package with sudo dpkg -i <package.deb> works fine. All seems fine when I perform the same actions on Ubuntu Bionic (18.04) There's nothing in the JFrog UI that indicates any issue with the package. How can I figure out what's going wrong? [0] https://github.com/datastax/cpp-driver cd to 'packaging' and run build_deb.sh
Hello I came across the same issue and found out the problem, since ubuntu 21.10 the default package compression method has changed to zstd but artifactory had no support of this until recently: RTFAC-19541. So you can either update your artifactory instance to at least version 7.38.0 or use another compression method in your debian package generation process, see the this answer. Too bad that artifactory didn't print any error for this.
Can't brew install MariaDB 5.5 on Mac OS
does anyone know how I could install MariaDB 5.5 server with Homebrew on Mac OS? I've tried running brew install mariadb#5.5 but it doesn’t work.
You can also Try Building it from source! you can use Homebrew to build MariaDB from source.Its helpful if you want to use a different version of the server or enable some different capabilities that are not included in the bottle package or if you are having issues with the bottle package. To build MariaDB Server with these engines, you must first install boost and judy. And please can you be specific about the errors you encountered. You might want to check this site:
How can I compile RpostgreSQL with libssl and libpg and SSL activation
I am using R on Windows to connect to a PostgreSQL database hosted on AWS. The database is set up using forcessl = 1 - this means that any connection needs to be set up with sslmode=require.
The base RPostgreSQL package does not provide any exposure to ssl options. This has been raised as an issue many times (see here, here, here and here)
I know there is a workaround using the RPostgres package, but for other functionality reasons I would much prefer to use the RPostgreSQL package.
A few answers (e.g. here) have proposed using a modified dbname to connect with ssl like so:
dbConnect(dbDriver('PostgreSQL'),
dbname = 'dbname=foobar sslmode=require', # modified dbname
host = 'foobar.rds.amazonaws.com',
port = 5439,
user = 'foobar',
password = 'foobar')
But this did not work for me using the CRAN version of the package. This led me to a recent issue raised on the RPostgreSQL github: https://github.com/tomoakin/RPostgreSQL/issues/88
The initial user was able to use the modified dbname method when he compiled the package from source. On Windows, using the latest source package (0.6.2) compiled with RTools, I get the following error when I run the modified dbname code:
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect xxxxx.rds.amazonawss.com:5432 on dbname "xxxxxxx": sslmode value "require" invalid when SSL support is not compiled in
)
From this and the rest of the thread, it looks like SSL is not possible from current source in both Windows and Mac. However, the developer suggests:
If you compile in a environment where libssl and libpq was made SSL activated form, then the driver can use SSL.
I think this means I could manually download the libs and compile myself, but I am not sure if it is a quick fix or if it would require significant rewriting of the package. Any help or pointing in the right direction would be much appreciated. How can I do this in a safe, repeatable way?
I was able to solve this for the El Capitan macOS R users in my office, by doing the following:
Remove RPostgreSQL R package if you already have installed. Methods vary on how to do this, but from either R.app console or R in Terminal, type remove.packages('RPostgreSQL')
Make sure you have Homebrew installed, and from Terminal run: brew install libpq openssl
Open R.app, and from the Packages & Data menu, select Package Installer.
From the first drop-down menu, choose CRAN (sources) (choose mirror closest to you if you haven't used this before).
Using package search, find RPostgreSQL and for the options below, keep At System Level checked, and check Install Dependencies, then click Install Selected.
Quit out of all R and RStudio programs, and try using the new from source installed RPostgreSQL package.
DISCLAIMER: If you have heavy compile dependencies on OpenSSL or libpq for other programs, I have no idea how doing the above may break other programs.
Building on Windows is a can of worms. See R-Admin Windows Toolset. The only openSSL binaries for windows are from unknown developers. Building and installing openSSL on windows is another can of worms that you will need to research. It might be easier to install openSSL inside the R Windows build environment, but I have no experience with that.
EDIT: It turns out that when installing postgres on Windows, postgres installs openSSL. That means that the central problem on Windows is installing the Windows Toolset for R, installing postgres, then pointing the R build system to libpq.
Another solution would be to run linux in a virtual machine under windows. Here is one way to Install linux on Windows. With linux, depending on the distribution, you would only need to do something like the following from the linux command line (for a RedHat variant of linux):
sudo yum install openSSL
sudo yum install postgresql96
sudo yum install R
Line 2 installs libpq which is required for RPostgreSQL. It is libpq which must be compiled with openSSL. You will only be installing and using the PostgreSQL client, not the server and will also get psql. There might be other packages required, see R linux toolset. Normally, these will get pulled in with the above and should not be a problem.
RPostgreSQL contains a version of libpq, but the compile script does not look like it checks for openSSL, at least not on macOS. So it is important to get a system provided libpq installed.
It is also important for the RPostgreSQL configure script to find pg_config, which is installed when postgres client is installed. Not sure about windows through. So make sure pg_config is in your path. Type pg_config to find out.
Now you need to download and compile RPostgreSQL. To start R, type the following at the linux terminal.
R
Then from within R, get, compile and install RPostgreSQL:
install.packages("https://cran.r-project.org/src/contrib/RPostgreSQL_0.6-2.tar.gz", repo=NULL, type="source")
This should compile and install this version of RPostgreSQL. This last line should also work in windows if you have the windows tools installed correctly.
Hopefully, this gives you some ideas.
How to specify include and lib directories when locally installing RODBC?
i am trying to install RODBC with the commadn R CMD INSTALL -l /my/local/path RODBC.tar.gz and it wont find sql.h and sqlext.h ..how do i pass the include and lib paths to this command?
I had a similar problem recently while trying to install RODBC on an instance of Centos 5.8 x64. (Instead of using R CMD install, I just did sudo R, then installed the package inside R - but I was still having the same problem).
I resolved this by installing the following packages using yum:
unixODBC
unixODBC-libs
unixODBC-devel
mysql-connector-odbc
freetds
freetds-devel
Those last two are necessary because I was interfacing with an MSSQL server, which requires TDS. Anyway, once I installed unixODBC and the libs and development package, I was able to install RODBC with no difficulty (again, using sudo R and then install.packages("RODBC").
The only other thing to do is install the correct drivers/libraries for ODBC, which is what mysql-connector-odbc is. You will need to configure unixODBC to meet your needs, but the documentation is pretty solid, so I don't think you'll have too much difficulty.