AWS RedShift - .NET Core (ODBC Support?) - odbc

Question

How can I connect and run queries against AWS RedShift using .NET Core? Code sample please. I have gone through the AWS docs and .Net Core docs but no luck.

Answer 1

This answer is one for a particular point in time and won't age well...
The EntityFramework Core project is the one I'd keep the closest eye on. The lack of ODBC is well known, especially for those who want to connect to Oracle. For the time being, you may need to fork an Oracle client for .NET core and make modifications as necessary.
I found these projects after a quick Google search that may be able to help you for now...
- https://github.com/LinqDan/oracleclientcore
- https://github.com/LinqDan/Mono.Data.OdbcCore
Longer term, you'll want to keep an eye on these two GitHub issues which are tracking it for EntityFramework Core and the .NETStandard APIs..
https://github.com/dotnet/corefx/issues/13035
https://github.com/aspnet/EntityFramework/issues/7432
Update 6/23/2017:
This is now possible through the Npgsql.EntityFrameworkCore.PostgreSQL NuGet package and associated Entity Framework Core packages. Apparently the PostgreSQL team was tired of waiting around for ODBC support (which still isn't available in the latest netstandard2.0 yet) and wrote their own driver using netstandard - back in the November timeframe. The getting started page on the npgsql website covers it's usage in the old JSON project format - but the dependencies listed are still valid.
Here is how you would use the package...
using (var conn = new NpgsqlConnection("Host=myserver;Username=mylogin;Password=******;Database=music"))
{
conn.Open();
using (var cmd = new NpgsqlCommand())
{
cmd.Connection = conn;
cmd.CommandText = "SELECT name FROM artists";
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
}
}
}
}
One thing to keep in mind when using this driver - it's written for PostgreSQL, not Redshift. While Redshift is based on PostgreSQL, it's underlying engine is much more like Cassandra than anything else. As a result, Amazon had to make some choices in the development to drop certain things that PostgreSQL does support - such as SQL variables. Because of this, you will have a fairly limiting experience for certain things that you might be used to in other Entity Framework implementations. As long as you stay with using the direct access *Connection, *Command, and DataReader classes and write your own SQL, you should be fine.

Answer 2

At the time of writing this answer there are ODBC drivers readily available for Redshift.
For Windows, you can find the "Amazon Redshift" driver installation file here:
https://docs.aws.amazon.com/redshift/latest/mgmt/install-odbc-driver-windows.html
This will need to be installed on the machine (or machines) that run the application required to connect to Redshift. For automated deployments, it would be a good idea to have a powershell script (or MSBuild step) that checks if it is installed on the production machine and install it if it doesn't exist.
For Linux, the driver can be found here:
https://docs.aws.amazon.com/redshift/latest/mgmt/install-odbc-driver-linux.html
Since most of the world of .NET Core is moving towards using Linux docker containers I will elaborate further on how to set this up in a docker environment using a Dockerfile.
First you will need to create a shell script (e.g. install-driver.sh) that does an apt-get for the relevant odbc driver you want from the link above.
Something like this:
#!/bin/bash
# Install the Redshift ODBC driver manager
apt-get update \
&& apt-get install -y --no-install-recommends unixodbc
if ! curl -s https://s3.amazonaws.com/redshift-downloads/drivers/odbc/[latestversion].deb -o driver.deb; then
echo 'Failed to download Redshift ODBC Driver!' 1>&2
exit 1
fi
# Install the Redshift ODBC driver
apt install ./driver.deb
There are also three files required for configuring the Amazon Redshift ODBC driver on Linux: amazon.redshiftodbc.ini, odbc.ini, and odbcinst.ini. This is explained here:
https://docs.aws.amazon.com/redshift/latest/mgmt/odbc-driver-configure-linux-mac.html
As mentioned in that link, you will have to add the following lines to your .sh file:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export ODBCINI=/etc/odbc.ini
export AMAZONREDSHIFTODBCINI=/etc/amazon.redshiftodbc.ini
export ODBCSYSINI=/usr/local/odbc
Once you have created the .sh and .ini files put them into a common directory (e.g. /odbc/) and include them as part of your Dockerfile commands. Something like this:
FROM microsoft/dotnet:2.1-sdk AS base
WORKDIR /app # or wherever your working directory resides
COPY ./odbc/odbc.ini /etc/odbc.ini
COPY ./odbc/odbcinst.ini /etc/odbcinst.ini
COPY ./odbc/amazon.redshiftodbc.ini /etc/amazon.redshiftodbc.ini
COPY ./odbc/install-driver.sh /tmp/install-driver.sh
RUN chmod +x /tmp/install-driver.sh
RUN /tmp/install-driver.sh
# dotnet restore, build, publish and ENTRYPOINT commands here...
That should be enough for you to run the docker build and docker run commands with the Redshift driver installed within your docker container.
It means that your application code can use OdbcConnection just like any other database driver:
using System;
using System.Data;
using System.Data.Odbc;
// field within class... value should be set from config
private string redshiftConnectionString = "Driver={Amazon Redshift (x64)}; Server=redshiftclusterhostname.region.redshift.amazonaws.com; Database=database; UID=user; PWD=password; Port=5439"
public DataSet ExecuteQuery(string query)
{
var dataSet = new DataSet();
using (var connection = new OdbcConnection(this.redshiftConnectionString))
{
var adapter = new OdbcDataAdapter(query, connection);
adapter.Fill(dataSet);
}
return dataSet;
}
Take note that the Driver= value should be the name of the driver you chose to download and install.

Answer 3

Simply connect to Amazon Redshift via a JDBC or ODBC Driver and access it like a normal SQL database.
You'll use the AWS API to start/stop the cluster, but all queries and requests go via the SQL connection.