I want to host Shiny applications on my company network using Docker for Windows.
How do I set up the Docker, odbc.ini, odbcinst.ini, freetds.conf, or possibly other files so that my Shiny application can query data from an internal Microsoft SQL Server (2016) database? The database server is not running on the same machine running the Docker container.
I don't know if I need a newer version of FreeTDS or if I have mis-configured one of the files. I tried using the IP address to the server instead of sql-server.host.com in all files but get the same error message below.
$ tsql -C output:
Compile-time settings (established with the "configure" script)
Version: freetds v1.00.104
freetds.conf directory: /etc/freetds
MS db-lib source compatibility: no
Sybase binary compatibility: yes
Thread safety: yes
iconv library: yes
TDS version: 4.2
iODBC: no
unixodbc: yes
SSPI "trusted" logins: no
Kerberos: yes
OpenSSL: no
GnuTLS: yes
MARS: no
$ odbcinst -j output:
unixODBC 2.3.6
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /root/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
$ cat etc/odbcinst.ini output:
[FreeTDS]
Description = FreeTDS unixODBC Driver
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
$ cat etc/odbc.ini output:
[sql-server]
driver = FreeTDS
server = sql-server.host.com
port = 1433
TDS_Version = 4.2
$ cat etc/freetds/freetds.conf output:
[sql-server]
host = sql-server.host.com
port = 1433
tds version = 4.2
Command in R giving error:
con <- dbConnect(odbc::odbc(),
driver = "FreeTDS",
server = "sql-server.host.com",
port = 1433,
database = "database name",
TDS_Version = 4.2)
Error:
Error: nanodbc/nanodbc.cpp:950: 08001: [FreeTDS][SQL Server]Unable to connect to data source
Execution halted
Docker file:
# Install R version 3.5.3
FROM r-base:3.5.3
# Install Ubuntu packages
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev/unstable \
libxt-dev \
libssl-dev \
unixodbc unixodbc-dev \
freetds-bin freetds-dev tdsodbc
# Edit odbc.ini, odbcinst.ini, and freetds.conf files
RUN echo "[sql-server]\n\
host = sql-server.host.com\n\
port = 1433\n\
tds version = 4.2" >> /etc/freetds.conf
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so" >> /etc/odbcinst.ini
RUN echo "[sql-server]\n\
driver = FreeTDS\n\
server = sql-server.host.com\n\
port = 1433\n\
TDS_Version = 4.2" >> /etc/odbc.ini
# Install R packages that are required
RUN R -e "install.packages(c('shiny', 'DBI', 'odbc'), repos='http://cran.rstudio.com/')"
# copy the app to the image
RUN mkdir /root/shiny_example
COPY app /root/shiny_example
COPY Rprofile.site /usr/lib/R/etc/
# Make the ShinyApp available at port 801
EXPOSE 801
CMD ["R", "-e", "shiny::runApp('/root/shiny_example')"]
Docker build and run commands:
docker build . -t shiny_example
docker run -it --network=host -p 801:801 shiny_example
Note that following R code works on my Windows machine running the Docker container and I can successfully query the database:
library(DBI)
con <- dbConnect(odbc::odbc(),
driver = "SQL server",
server = "sql-server.host.com")
$ isql -v sql-server output:
[S1000][unixODBC][FreeTDS][SQL Server]Unable to connect to data source
[01000][unixODBC][FreeTDS][SQL Server]Unknown host machine name.
[ISQL]ERROR: Could not SQLConnect
$ tsql -S sql-server output:
locale is "en_US.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
Error 20013 (severity 2):
Unknown host machine name.
There was a problem connecting to the server
It looks like you're correct, but might have missed a little. I had similar issue, but I was able to fix it!
On python+mssqlserver+pymssql+docker(unbuntu16.04 base image)
Without fixing the end, upon running my code(using pymssql) was giving me this error
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "src/pymssql.pyx", line 645, in pymssql.connect
pymssql.InterfaceError: Connection to the database failed for an unknown reason.
I followed created 3 files and copied in the image!
Use myserver.orgName.com where MSsql server is hosted.
docker/odbcinst.ini
[FreeTDS]
Description = v0.91 with protocol v7.3
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
docker/odbc.ini
[myserverdsn]
Driver = FreeTDS
Server = myserver.orgName.com
Port = 1433
TDS_Version = 7.3
docker/freetds.conf
[global]
# TDS protocol version, use:
# 7.3 for SQL Server 2008 or greater (tested through 2014)
# 7.2 for SQL Server 2005
# 7.1 for SQL Server 2000
# 7.0 for SQL Server 7
tds version = 7.2
port = 1433
# Whether to write a TDSDUMP file for diagnostic purposes
# (setting this to /tmp is insecure on a multi-user system)
; dump file = /tmp/freetds.log
; debug flags = 0xffff
# Command and connection timeouts
; timeout = 10
; connect timeout = 10
# If you get out-of-memory errors, it may mean that your client
# is trying to allocate a huge buffer for a TEXT field.
# Try setting 'text size' to a more reasonable limit
text size = 64512
# A typical Microsoft server
[myserverdsn]
host = myserver.orgName.com
port = 1433
tds version = 7.3
Dockerfile content
RUN apt-get -y install unixodbc unixodbc-dev freetds-dev freetds-bin tdsodbc
COPY freetds.conf /etc/freetds/freetds.conf
COPY odbc.ini /etc/odbc.ini
COPY odbcinst.ini /etc/odbcinst.ini
Test python code that works:
python
>>> import pymssql
>>> conn = pymssql.connect(server = 'myserver.orgName.com',
user = 'myusername',
password = 'mypassword',
database= 'mydbname')
works without error!
Related
I want run dask on EMR using YarnCluster.
I have used below bootstrap script but I have run these instructions in SSH console.
#!/bin/bash
HELP="Usage: bootstrap-dask [OPTIONS]
Example AWS EMR Bootstrap Action to install and configure Dask and Jupyter
By default it does the following things:
- Installs miniconda
- Installs dask, distributed, dask-yarn, pyarrow, and s3fs. This list can be
extended using the --conda-packages flag below.
- Packages this environment for distribution to the workers.
- Installs and starts a jupyter notebook server running on port 8888. This can
be disabled with the --no-jupyter flag below.
Options:
--jupyter / --no-jupyter Whether to also install and start a Jupyter
Notebook Server. Default is True.
--password, -pw Set the password for the Jupyter Notebook
Server. Default is 'dask-user'.
--conda-packages Extra packages to install from conda.
"
# Parse Inputs. This is specific to this script, and can be ignored
# -----------------------------------------------------------------
# -----------------------------------------------------------------------------
# 1. Check if running on the master node. If not, there's nothing do.
# -----------------------------------------------------------------------------
grep -q '"isMaster": true' /mnt/var/lib/info/instance.json \
|| { echo "Not running on master node, nothing to do" && exit 0; }
# -----------------------------------------------------------------------------
# 2. Install Miniconda
# -----------------------------------------------------------------------------
echo "Installing Miniconda"
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh
bash /tmp/miniconda.sh -b -p $HOME/miniconda
rm /tmp/miniconda.sh
echo -e '\nexport PATH=$HOME/miniconda/bin:$PATH' >> $HOME/.bashrc
source $HOME/.bashrc
conda update conda -y
# configure conda environment
#source ~/miniconda/etc/profile.d/conda.sh
#conda activate base
# -----------------------------------------------------------------------------
# 3. Install packages to use in packaged environment
#
# We install a few packages by default, and allow users to extend this list
# with a CLI flag:
#
# - dask-yarn >= 0.7.0, for deploying Dask on YARN.
# - pyarrow for working with hdfs, parquet, ORC, etc...
# - s3fs for access to s3
# - conda-pack for packaging the environment for distribution
# - ensure tornado 5, since tornado 6 doesn't work with jupyter-server-proxy
# -----------------------------------------------------------------------------
echo "Installing base packages"
conda install \
-c conda-forge \
-y \
-q \
dask-yarn \
s3fs \
conda-pack \
tornado
pip3 install pyarrow
# -----------------------------------------------------------------------------
# 4. Package the environment to be distributed to worker nodes
# -----------------------------------------------------------------------------
echo "Packaging environment"
conda pack -q -o $HOME/environment.tar.gz
# -----------------------------------------------------------------------------
# 5. List all packages in the worker environment
# -----------------------------------------------------------------------------
echo "Packages installed in the worker environment:"
conda list
# -----------------------------------------------------------------------------
# 6. Configure Dask
#
# This isn't necessary, but for this particular bootstrap script it will make a
# few things easier:
#
# - Configure the cluster's dashboard link to show the proxied version through
# jupyter-server-proxy. This allows access to the dashboard with only an ssh
# tunnel to the notebook.
#
# - Specify the pre-packaged python environment, so users don't have to
#
# - Set the default deploy-mode to local, so the dashboard proxying works
#
# - Specify the location of the native libhdfs library so pyarrow can find it
# on the workers and the client (if submitting applications).
# ------------------------------------------------------------------------------
echo "Configuring Dask"
mkdir -p $HOME/.config/dask
cat <<EOT >> $HOME/.config/dask/config.yaml
distributed:
dashboard:
link: "/proxy/{port}/status"
yarn:
environment: /home/hadoop/environment.tar.gz
deploy-mode: local
worker:
env:
ARROW_LIBHDFS_DIR: /usr/lib/hadoop/lib/native/
client:
env:
ARROW_LIBHDFS_DIR: /usr/lib/hadoop/lib/native/
EOT
# Also set ARROW_LIBHDFS_DIR in ~/.bashrc so it's set for the local user
echo -e '\nexport ARROW_LIBHDFS_DIR=/usr/lib/hadoop/lib/native' >> $HOME/.bashrc
# -----------------------------------------------------------------------------
# 8. Install jupyter notebook server and dependencies
#
# We do this after packaging the worker environments to keep the tar.gz as
# small as possible.
#
# We install the following packages:
#
# - notebook: the Jupyter Notebook Server
# - ipywidgets: used to provide an interactive UI for the YarnCluster objects
# - jupyter-server-proxy: used to proxy the dask dashboard through the notebook server
# -----------------------------------------------------------------------------
echo "Installing Jupyter"
conda install \
-c conda-forge \
-y \
-q \
notebook \
ipywidgets \
jupyter-server-proxy \
jupyter
# -----------------------------------------------------------------------------
# 9. List all packages in the client environment
# -----------------------------------------------------------------------------
echo "Packages installed in the client environment:"
conda list
# -----------------------------------------------------------------------------
# 10. Configure Jupyter Notebook
# -----------------------------------------------------------------------------
echo "Configuring Jupyter"
mkdir -p $HOME/.jupyter
JUPYTER_PASSWORD="dask-user"
HASHED_PASSWORD=`python -c "from notebook.auth import passwd; print(passwd('$JUPYTER_PASSWORD'))"`
cat <<EOF >> $HOME/.jupyter/jupyter_notebook_config.py
c.NotebookApp.password = u'$HASHED_PASSWORD'
c.NotebookApp.open_browser = False
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.port = 8888
EOF
# -----------------------------------------------------------------------------
# 11. Define an upstart service for the Jupyter Notebook Server
#
# This sets the notebook server up to properly run as a background service.
# -----------------------------------------------------------------------------
echo "Configuring Jupyter Notebook Upstart Service"
cat <<EOF > /tmp/jupyter-notebook.service
[Unit]
Description=Jupyter Notebook
[Service]
ExecStart=$HOME/miniconda/bin/jupyter-notebook --allow-root --config=$HOME/.jupyter/jupyter_notebook_config.py
Type=simple
PIDFile=/run/jupyter.pid
WorkingDirectory=$HOME
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/jupyter-notebook.service /etc/systemd/system/
sudo systemctl enable jupyter-notebook.service
# -----------------------------------------------------------------------------
# 12. Start the Jupyter Notebook Server
# -----------------------------------------------------------------------------
echo "Starting Jupyter Notebook Server"
sudo systemctl daemon-reload
sudo systemctl restart jupyter-notebook.service
#$HOME/miniconda/bin/jupyter-notebook --allow-root --config=$HOME/.jupyter/jupyter_notebook_config.py
after this i start jupyter notebook using $HOME/miniconda/bin/jupyter-notebook --allow-root --config=$HOME/.jupyter/jupyter_notebook_config.py
jupyter notebook start successfully.
When i run this code on notebook
from dask_yarn import YarnCluster
from dask.distributed import Client
# Create a cluster
cluster = YarnCluster()
# Connect to the cluster
client = Client(cluster)
it gives error like
AttributeError Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 client = Client(cluster)
File ~/miniconda/lib/python3.9/site-packages/distributed/client.py:835, in Client.__init__(self, address, loop, timeout, set_as_default, scheduler_file, security, asynchronous, name, heartbeat_interval, serializers, deserializers, extensions, direct_to_workers, connection_limit, **kwargs)
832 elif isinstance(getattr(address, "scheduler_address", None), str):
833 # It's a LocalCluster or LocalCluster-compatible object
834 self.cluster = address
--> 835 status = getattr(self.cluster, "status")
836 if status and status in [Status.closed, Status.closing]:
837 raise RuntimeError(
838 f"Trying to connect to an already closed or closing Cluster {self.cluster}."
839 )
AttributeError: 'YarnCluster' object has no attribute 'status'
Also when I use LocalCluster instead of YarnCluster it run perfectly. I am stuck here for days please help. Also how we configure worker nodes.
I am trying to connect to a Microsoft SQL Server database from R studio.
I installed the unixODBC driver manager using homebrew:
brew install unixodbc
I then tried to install the freetds driver as recommended on https://db.rstudio.com/best-practices/drivers/
# SQL Server ODBC Drivers (Free TDS)
brew install freetds --with-unixodbc
I received the error:
Error: invalid option: --with-unixodbc
I found a source saying the option is no longer available because it is default now. Thus, I ran:
brew install freetds
In R Studio (and R) I then attempted to list the installed driver, but it was not found:
> odbc::odbcListDrivers()
[1] name attribute value
<0 rows> (or 0-length row.names)
Next I tried to install the Microsoft ODBC driver for SQL Server (macOS) according to https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15 and as recommended on https://db.rstudio.com/databases/microsoft-sql-server/
I ran:
brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
brew update
HOMEBREW_NO_ENV_FILTERING=1 ACCEPT_EULA=Y brew install msodbcsql17 mssql-tools
To check that this driver was registered in an odbcinst.ini file I first ran:
odbcinst -j
with result:
unixODBC 2.3.9
DRIVERS............: /opt/homebrew/etc/odbcinst.ini
SYSTEM DATA SOURCES: /opt/homebrew/etc/odbc.ini
FILE DATA SOURCES..: /opt/homebrew/etc/ODBCDataSources
USER DATA SOURCES..: /Users/Gina/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
The driver was already registered in /opt/homebrew/etc/odbcinst.ini
view /opt/homebrew/etc/odbcinst.ini
file contents (note the freetds driver does not appear to be present):
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/homebrew/lib/libmsodbcsql.17.dylib
UsageCount=1
In Rstudio, I tried again to list the drivers and none were found:
> odbc::odbcListDrivers()
[1] name attribute value
<0 rows> (or 0-length row.names)
I have tried many more things, including:
copying the contents of /opt/homebrew/etc/odbcinst.ini to the user level file /Users/Gina/.odbcinst.ini
editing the /opt/homebrew/etc/odbc.ini file (which was previously empty) to include connection details (though I was planning to just specify these in the DBI::dbConnect command). New redacted file contents:
[ODBC Driver 17 for SQL Server]
Driver = ODBC Driver 17 for SQL Server
Servername = servername.net
Database = databasename
UserName = rstudioapi::askForPassword("Database user")
Password = rstudioapi::askForPassword("Database password")
Port = 1433
copying the contents of the /opt/homebrew/etc/odbc.ini to the user level file /Gina/.odbc.ini
setting an environmental variable:
export ODBCSYSINI=/opt/homebrew/etc
Note that when I run:
odbcinst -q -s
The driver is found:
[ODBC Driver 17 for SQL Server]
Thus, it appears that odbc::odbcListDrivers() should be able to find the driver. Any thoughts or suggestions would be very much appreciated! Thank you!
EDIT:
I also tried using the New Connection button and interestingly, my driver is listed there!
New Connection Window
However, when I click on it and click Test, I get an error:
Error: nanodbc/nanodbc.cpp:1021: 00000: [unixODBC][Driver Manager]Data source name not found and no default driver specified
Driver Connection Window with Error
I get the same error when I try:
isql -v ODBC Driver 17 for SQL Server myusername mypassword
[IM002][unixODBC][Driver Manager]Data source name not found and no
default driver specified
EDIT 2:
I changed the Data Source Name (top line in odbc.ini file) to "SQLSever" and changed the Servername argument to Server (I did this in both the system level file and the user level file). Now when I run the below command it tries to connect (though hits a new error).
isql -v SQLServer myusername mypassword
[08001][unixODBC][Microsoft][ODBC Driver 17 for SQL Server]SSL
Provider: [OpenSSL library could not be loaded, make sure OpenSSL
1.0 or 1.1 is installed]
[08001][unixODBC][Microsoft][ODBC Driver 17 for SQL Server]Client
unable to establish connection
[ISQL]ERROR: Could not SQLConnect
However, unfortunately, I still can not connect in R. I get the same errors as before.
Note that it appears that the isql command uses the system level files and RStudio uses the user level files. The files are identical in both places.
I read in a post on a similar issue that the M1 Macs have this problem but non-M1 Macs do not. Thus, I tried repeating the whole process on an older Macbook Pro and did not encounter the problem! odbc::odbcListDrivers() immediately listed the driver after installing the Microsoft ODBC driver for SQL Server (macOS). I did not need to edit or create any files.
With M1 Mac I was able to connect to SQL Server through terminal by changing the openssl folder to an older version. The driver installs openssl#3 (opt/homebrew/opt/openssl#3) but you actually need to use openssl#1.1 instead.
Here are the steps i followed in terminal:
brew install openssl#1.1
rm /opt/homebrew/opt/openssl
ln -s /opt/homebrew/Cellar/openssl#1.1/1.1.1l_1 /opt/homebrew/opt/openssl
This will create a symlink "openssl" and point it to the correct file (1.1.1l_1) inside the opt/homebrew/Cellar/openssl#1.1 folder.
Before creating the symlink verify the file name 1.1.1l_1 has not changed with a newer version in opt/homebrew/Cellar/openssl#1.1/1.1.1l_1
This solution worked for me:
Create two symlinks:
sudo mkdir -p /usr/local/etc
sudo ln -s /opt/homebrew/etc/odbc.ini /usr/local/etc/odbc.ini
sudo ln -s /opt/homebrew/etc/odbcinst.ini /usr/local/etc/odbcinst.ini
When I build and then run a Docker container which runs rstudio on Ubuntu, the odbc connection does not work when I add the odbc.ini file during the build. However, if I leave out the odbc.ini file from the build and instead add it myself from within the running container, the connection does indeed work.
So my problem is that I am trying to get the odbc connection up and running out of the box whenever this image is run, without the additional step of having to login to the ubuntu container instance and add connection details to the odbc.ini file.
Here's what the odbc.ini file looks like, with dummy data:
[PostgreSQL ANSI]
Driver = PostgreSQL ANSI
Database = GoogleData
Servername = somename.postgres.database.azure.com
UserName = docker_rstudio#somename
Password = abc123abc
Port = 5432
sslmode = require
I have a copy of this file, odbc.ini, in my repo directory and then include it in the build. My DockerFile.
FROM rocker/tidyverse:3.6.3
ENV ADD=SHINY
ENV ROOT=TRUE
ENV PASSWORD='abc123'
RUN apt-get update && apt-get install -y \
less \
vim \
unixodbc unixodbc-dev \
odbc-postgresql
ADD odbc.ini /etc/odbc.ini
ADD install_packages.R /tmp/install_packages.R
RUN Rscript /tmp/install_packages.R && rm -R /tmp/*
ADD flagship_ecommerce /home/rstudio/blah/zprojects/flagship_ecommerce
ADD commission_junction /home/rstudio/blah/zprojects/commission_junction
RUN mkdir /srv/shiny-server; ln -s /home/rstudio/blah/zprojects/ /srv/shiny-server/
If I then login to the instance via rstudio, the connection does not work, I get this error message:
Error: nanodbc/nanodbc.cpp:983: 00000: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
If I take a look at the file with less /etc/odbc.ini I do indeed see the connection details per my top code block.
If I then copy to home with cp /etc/odbc.ini /home/rstudio/.odbc.ini then, after that, my connection does work.
But, even if I amend my dockerfile with ADD odbc.ini /home/rstudio/.odbc.ini, the connection doesn't work. It only works when I manually add to /home/rstudio/.odbc.ini.
So my problem is two fold:
No matter what I try I cannot get /etc/odbc.ini to be detected by ubuntu to use as odbc connection string. Whether via Dockerfile or by manually adding it. I would prefer this since I want to connection to be available to anyone using the container.
I am able to get a connection when I manually copy whats in odbc.ini above to /home/rstudio/.odbc.ini however if I try to do this via the docker build, the connection does not work. I do see the file there. It exists with all the correct data, it is just not detected by odbc.
In case it's relevant:
odbcinst -j
unixODBC 2.3.6
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /home/rstudio/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
I believe the problem is with the format of your /etc/odbc.ini. I don't have all your scripts, but this is the Dockerfile I used:
FROM rocker/tidyverse:3.6.3
ENV ADD=SHINY
ENV ROOT=TRUE
ENV PASSWORD='abc123'
RUN apt-get update && apt-get install -y \
less \
vim \
unixodbc unixodbc-dev \
odbc-postgresql
RUN Rscript -e 'install.packages(c("DBI","odbc"))'
ADD ./odbc.ini /etc/odbc.ini
If I use an odbc.ini of this:
[mydb]
Driver = PostgreSQL ANSI
ServerName = 127.0.0.1
UserName = postgres
Password = mysecretpassword
Port = 35432
I see this (docker build and R startup messages truncated):
$ docker build -t quux2 .
$ docker run --net='host' -it --rm quux2 bash
> con <- DBI::dbConnect(odbc::odbc(), "mydb")
Error: nanodbc/nanodbc.cpp:983: 00000: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
When I changed the indentation of the file to this:
[mydb]
Driver = PostgreSQL ANSI
ServerName = 127.0.0.1
UserName = postgres
Password = mysecretpassword
Port = 35432
I see this:
$ docker build -t quux3 .
$ docker run --net='host' -it --rm quux3 bash
> con <- DBI::dbConnect(odbc::odbc(), "mydb")
> DBI::dbGetQuery(con, "select 1 as a")
a
1 1
(For this demonstration, I'm running postgres:11 as another container, but I don't think that that's relevant, it's the indented values.)
I am no expert in docker, and have failed to find the specific documentation for this. But from experience it seems that every time you add a new layer (eg. using RUN) any copy from previous layers are "forgotten" (Note: this might be completely wrong, if so please someone correct me and specify the documentation).
So I would try to combine your RUN arguments and add every file right before the RUN statement they're needed. This has the added benefit of reducing the final image size, because of the way layers are created and kept.
FROM rocker/tidyverse:3.6.3
ENV ADD=SHINY
ENV ROOT=TRUE
ENV PASSWORD='abc123'
#add files (could also combine them into a single tar file and add it. Or add it via git, which is often used)
ADD odbc.ini /etc/odbc.ini
ADD install_packages.R /tmp/install_packages.R
ADD flagship_ecommerce /home/rstudio/blah/zprojects/flagship_ecommerce
ADD commission_junction /home/rstudio/blah/zprojects/commission_junction
#Combine all runs into a single statement
RUN apt-get update && apt-get install -y \
less \
vim \
unixodbc unixodbc-dev \
odbc-postgresql \
&& Rscript /tmp/install_packages.R \
&& rm -R /tmp/* \
&& mkdir /srv/shiny-server \
&& ln -s /home/rstudio/blah/zprojects/ /srv/shiny-server/
Note that now add technically comes right before the statement where it is used.
Trying to use sqlcmd from mssql-tools package (loosly following the docs here: https://github.com/mkleehammer/pyodbc/wiki/Connecting-to-SQL-Server-from-RHEL-6-or-Centos-7) and seeing error
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Data source name not found and no default driver specified.
even though I believe that both DSN and driver are specified (see below)
[airflow#airflowetl ~]$
[airflow#airflowetl ~]$
[airflow#airflowetl ~]$ # trying to connect to DB via DSN (-S option)
[airflow#airflowetl ~]$ /opt/mssql-tools/bin/sqlcmd -Q "select count(*) from MYTABLE" -D -S "MyMSSQLServer" -U "myuser" -P "mypass" -d mydb -W -h-1 -k
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Data source name not found and no default driver specified.
[airflow#airflowetl ~]$
[airflow#airflowetl ~]$
[airflow#airflowetl ~]$ # checking odbc config files
[airflow#airflowetl ~]# odbcinst -j
unixODBC 2.3.7
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /root/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
[airflow#airflowetl ~]$ cat /etc/odbc.ini
[MyMSSQLServer]
Driver=ODBC Driver 17 for SQL Server
Description=My MS SQL Server
Trace=No
Server=172.18.1.23
[airflow#airflowetl ~]$ cat /etc/odbcinst.ini
[airflow#airflowetl ~]$ ls -lha /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.4.so.2.1
-rwxr-xr-x 1 root root 2.0M Sep 26 06:19 /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.4.so.2.1
Notice that /etc/odbcinst.ini is blank when (I think) it should corespond to Driver entries in the odbc.ini file. Should I just fill this in manually? Why does it not auto-fill in this case (as it has done on other machines where I have set up odbc)?
Not much experience with ODBC setup, so not sure what to make of this error. Anyone with more experience have any fuirther debugging tips or know how to fix?
Did you install the driver? Please have a look this link and follow the steps.
https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017
Regards
xavy
I want to upgrade the postgresql 9.1 database to postgresql 11. First I want to get the data on the remote pg9.1 server with pg_dump from the pg11 server to make an experiment. How do I copy the database on the remote server to the new server with the new pg_dump.
The following code is using old pg_dump and it doesn't work.
ssh user # remote_machine "pg_dump -U dbuser -h localhost -C --column-inserts" \ >> backup_file_on_your_local_machine.sql
pg11 on the network connected to the old database by taking full dump I solved
pg_dumpall -U postgres -h 10.100.80.100 -p 5432 --clean --file=/pg11localstoragepath/backup/mydb_backup.dump