I want to use pyodbc in aws qlue python shell but it require odbc driver. Currently I get error like "Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)"
Is there any way to install odbc driver into glue
I wanted to do the same, but there is no straight forward way it seems. I guess, you could do it by adding a driver to your self-built Python .wheel or do some kind of run-time downloading of a driver etc.
I can offer a simpler alternative though:
pymssql does exactly this for you. It's a Python package that comes with the FreeTDS mssql odbc driver included. So it's just a pip install pymssql to get you started. I've tested it successfully on a Glue Pythonshell Job. You'll just need to add the package to the --additional-python-modules parameter of your job, so that is becomes available. Keep in mind, that you might still need to create a Glue Connection and add it to your Job. Even though you will not use the connection directly, you'll need it for the network connectivity from within your Job to your DB-Server.
Related
I run BlacBox Component Builder under Wine/Ubuntu. I need to set up ODBC in order to access databases (PostgresQL in my case) in BBCB programs (Component Pascal). They have a unique and wonderful SQL access library that allows to accept SELECT results directly into a record variable (analogous to structures in the C world or class instances in C++). Such record is called an interactor. No preprocessor is used, and query parameters are taken directly out of program variables (Like so: "SELECT * FROM users WHERE name = :u.name AND age < :maxage", where u is a program variable of type RECORD and .name is it's field, maxage is also a program variable). Anyway, very convenient from a programmer's perspective.
However, BBCB does not have any native DB drivers for Linux, so I have to use the Windows BBCB version in Wine. Some years ago, I was able to set up unixodbc in Ubuntu, and unixodbc's DSNs became immediately available in Wine. Not this time around.
So, the question is, How do I set up ODBC in Ubuntu so that it's DSNs would be visible in Wine?
I found the trick!
BBCB is a 32-bit app, so obviousely it makes calls to 32-bit WINE/winapi functions, and they seemingly can't talk to the unixodbc/64bit.
The trick was to install :i386 versions of unixodbc and the DB odbc driver, then create a 32-bit wine prefix, and then run my app from there. In greater detail:
If not already done,
dpkg --add-architecture i386
sudo apt install unixodbc:i386
install database driver for odbc for your database server; in my case,
sudo apt install postgresql-odbc:i386
Assuming ~/.wine32 does not exist:
WINEPREFIX=~/.wine32 WINEARCH=win32 wine wineboot
(some manuals insist you have to have a wine 64-bit prefix before this)
Now copy BlackBox from it's previous location, or install anew, into the ~/.wine32 prefix, and from now on run BB as follows:
WINEPREFIX=~/.wine32 WINEARCH=win32 <program.exe-file-location>
Now BB's SqlOdbc module can perfectly see the DSNs I set up in unixodbc.
I got my connection from R to a MS SQL Server database working on a Mac. I was able to get the right driver for the connection string by issuing the odbcListDrivers() command. However, I don't know if it was there or not before I started installing things like freetds and brew install --no-sandbox msodbcsql17 mssql-tools.
What I'd like to know is: with the odbcListDrivers(), how do I know what the file is it pulls from (a .ini one, I'm thinking)? I struggled because the configuration files I configured for freetds and another odbc.ini file don't affect what R is reading I believe, and made troubleshooting difficult.
In case I need to configure more drivers or DSN (?) entries.
I am working on a machine without admin rights. I use sql developer to connect to an internal database. I would like to connect via R also.
Is there any way I can do this, without admin rights? Some solutions require me to set up a systemDNS - which I can not do.
Other requires me to install jvm.dll
My environment: Windows7, sqldeveloper, connection method is via TNS file.
Connecting to SQL Developer via R is far more difficult than other databases I've encountered. It's important that you have jdbc6.jar installed on your machine, and that you know the file path to where it was installed. Installing the jar file does not require admin rights. You can install the jar file from Oracle's website.
I use the RJDBC package to connect like so:
library(RJDBC)
jdbcDriver <- JDBC("oracle.jdbc.OracleDriver", classPath = "file path to where ojdbc6.jar is installed on your computer")
jdbcConnection <- dbConnect(jdbcDriver, "jdbc:oracle:thin:#YOUR_SERVER","YOUR_USERNAME","YOUR_PASSWORD")
You can then test the connection with a number of commands; I typically use:
dbListTables(jdbcConnection)
Another favorite of mine is to use dbplyr for dplyr-like functions when working with databases:
library(dbplyr)
tbl(jdbcConnection, "SAMPLE_TABLE_NAME")
The resulting output will be the data from the queried table in tibble form.
You can set the environment variables in your R session.
Sys.setenv(OCI_LIB64="/Path/to/instantclient",OCI_INC="/Path/to/instantclient/sdk/include")
You can put this in the file .Rprofile in your home directory, and RStudio will run it each time you begin a new session. Once you have this in .Rprofile you should be able to install ROracle.
So: I'm using Ubuntu 12.04 64 bit; I've set up unixODBC, and I'm trying to open an ODBC connection in Julia. I've tried this on both stable 0.20 and the latest nightly of Julia 0.30; I'm using ODBC.jl version 0.3.1. The problem I hit is that Julia seems to not be loading my DSNs. For instance:
ulia> ODBC.listdsns()
0x2 Array{String,2}
On the other hand, isql can connect to my DSNs just fine (so unixODBC is set up correctly). So... now I'm not sure how to proceed. I've tried using ODBC.advancedconnect(), but oddly, I get the same error message as when I try to use a DSN:
IM002: [unixODBC][Driver Manager]Data source name not found, and no default driver specified
There is a note about setting const odbc_dm on the ODBC.jl github page; I can't find any libs that sound like the one that note is talking about, but maybe that's the problem?
Any suggestions appreciated -- I don't particularly know how to proceed with this one, so this posting feels a little spare on details. If there's more information that would be helpful, just let me know and I'll get it posted.
ODBC.jl developer here. Yeah, it's probably because the package isn't finding your unixODBC library. Once you run using ODBC, type ODBC.odbc_dm to see what the package is picking up. You may have to manually set it (by modifying the ODBC_types.jl file in your package directory, or I can add some better logic to pick it up (better long-term). Feel free to open an issue at the repo and we can troubleshoot from there.
Installing the RODBC package on Ubuntu is a bit of a kludge. First I learned to install the following:
$ sudo apt-get install r-cran-rodbc
That wasn't good enough as the package was still looking for header files. I solved this issue by:
$ sudo apt-get install unixodbc-dev
Good, RODBC installed properly on the Ubuntu machine. But when I try to run the following script:
## import excel file from Dropbox
require("RODBC")
channel <- odbcConnectExcel("~/Dropbox/DATA/SAMPLE/petro.xls")
petro <- sqlFetch (channel, "weekly")
odbcClose(channel)
str(petro)
head(petro)
I get an error thrown that function odbcConnectExcel not found. I checked the case of each letter, making sure it was not a simple typo. Nope. Then I ran this same script on a Windows R installation (file path different, of course) and the script works.
Any idea of why Ubuntu R installation cannot find the odbcConnectExcel function and how I can get this to work?
That functionality is available where Excel is available. In other words: not on Ubuntu.
For reference, from the R Data Import / Export manual (with my highlighting):
4.3.2 Package RODBC
Package RODBC on CRAN provides an
interface to database sources
supporting an ODBC interface. This is
very widely available, and allows the
same R code to access different
database systems. RODBC runs on
Unix/Linux, Windows and Mac OS X, and
almost all database systems provide
support for ODBC. We have tested
Microsoft SQL Server, Access, MySQL,
PostgreSQL, Oracle and IBM DB2 on
Windows and MySQL, Oracle, PostgreSQL
and SQLite on Linux.
ODBC is a client-server system, and we
have happily connected to a DBMS
running on a Unix server from a
Windows client, and vice versa.
On Windows ODBC support is normally
installed, and current versions are
available from
http://www.microsoft.com/data/odbc/ as
part of MDAC. On Unix/Linux you will
need an ODBC Driver Manager such as
unixODBC (http://www.unixODBC.org) or
iOBDC (http://www.iODBC.org: this is
pre-installed in Mac OS X) and an
installed driver for your database
system.
Windows provides drivers not just for
DBMSs but also for Excel (.xls)
spreadsheets, DBase (.dbf) files and
even text files. (The named
applications do not need to be
installed. Which file formats are
supported depends on the the versions
of the drivers.) There are versions
for Excel 2007 and Access 2007 (go to
http://download.microsoft.com, and
search for Office ODBC, which will
lead to AccessDatabaseEngine.exe), the
`2007 Office System Driver'.
I've found RODBC to be a real pain in the Ubuntu. Maybe it's because I don't know the right incantations, but I switched to RJDBC and have had much better luck with it. As discussed here.
As Dirk says, that wont solve your Excel problem. For writing Excel I've had very good luck with the WriteXLS package. In Ubuntu I found it quite easy to set up. I had Perl and many of the packages already installed and had to simply install Text::CSV_XS which I installed with the GUI package manager. The reason I like WriteXLS is the ability to write data frames to different sheets in the Excel file. And now that I look at your question I see that you want to READ Excel files not WRITE them. Hell. WriteXLS doesn't do that. Stick with gdata, like Dirk said in his comments:
gdata on CRAN and you are going to want the read.xls() function:
read.xls("//path//to/excelfile.xls", sheet = 1, verbose=FALSE, pattern, ...,
method=c("csv","tsv","tab"), perl="perl")
you may need to run installXLSXsupport which installs the needed Perl modules.
read.xls expect sheet numbers, not names. The method parameter is simply the intermediate file format. If your data has tabs then don't use tab as the intermediate format. And likewise for commas and csv.