Has anyone been successfully able to run Snowflake & snowflake-connector-python on AirFlow?
I am trying to install it using these commands
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org snowflake-sqlalchemy
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org apache-airflow[snowflake]
But I am unable to get it working.
I don't see any errors/issues in the installation process but when I try to use Adhoc Query feature after establishing the connection, I don't see the conn_id for my snowflake connections. While I am able to see conn_ids for my other mssql connections.
Any help would be greatly appreciated.
Related
My attempts include:
[ here i have python 3.9 so my constraints is 3.9]
I created a virtual environment and attempted to install Apache-Airflow by using pip install 'apache-airflow==2.5.1' / —constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.1/constraints-3.9.txt," but I received an error that I couldn't understand, so I looked on the stack community for advice and learned that I needed to add -t, which I did.
but then I started getting permission errors, then I went into community posts and how to fix permission errors, and it was suggested that I should run command prompt as administrator and then install unfortunately this thing isn't working for me
I've tried running cmd as administrator, going to my project's directory, activating virtualenv, and installing the library, but I still get the same problem. PermissionError: [WinError 5] Access is denied: 'D:\\'
Thanks in advance.
Try removing the slash (/) from the command before the --constraint and then running inside the cmd terminal; it will work.
pip install apache-airflow==2.5.1 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.1/constraints-3.9.txt"
Note : Please be aware that the above recommendation is for Windows, but even if it is successfully installed you won't be able to run airflow because of the files used, such as pwd and others. Therefore, I will recommend that you use airflow in Windows using Ubuntu. You can follow this link to install Ubuntu in your system and set up airflow.
youtube-video-url
I can see the following in my pip list but when I try to add a Snowflake connection via the GUI, Snowflake is not an option from the dropdown.
apache-airflow-providers-snowflake 2.1.0
snowflake-connector-python 2.5.1
snowflake-sqlalchemy 1.2.3
Am I missing something?
I have had this issue with MWAA recently.
I find that if I select AWS in the drop down and provide the correct snowflake host name etc it works though.
I run into the same issue using the official helm chart 1.3.0.
But finally I was able to make Snowflake connection visible by doing the following steps:
I uninstalled the apache-airflow-providers-google. Not sure whether this is important, but I like to mention it here. I did this, because I got some warnings.
Because with SQLAlchemy 1.4 some breaking changes were introduced, I made sure that version 1.3.24 gets installed. Based on that I choosed the fitting version for snowflake stuff.
So this is my requirements.txt for my custom Airflow container:
apache-airflow-providers-snowflake==2.3.0
pyarrow==5.0.0
snowflake-connector-python==2.5.1
snowflake-sqlalchemy==1.2.5
SQLAlchemy==1.3.24
This is my Dockerfile:
FROM apache/airflow:2.2.1-python3.8
## adding missing python packages
USER airflow
COPY requirements.txt .
RUN pip uninstall apache-airflow-providers-google -y \
&& pip install -r requirements.txt
I had the same issue where my pip freeze showed the apache-airflow-providers-snowflake yet I did not have the provider in the UI. I had to add the line apache-airflow-providers-snowflake to my requirements.txt file and then restart. Then I was able to see the Snowflake provider and connector in the UI.
My OS is Mac OS. I followed airflow official installation guide to install. But when I test: airflow test tutorial print_date 2015-06-01 from airflow testing it doesn't print any output. The result is here.. I wonder did I install it successfully? I've run other commands on the official airflow testing page. They report no error.
So far I see only WARNING output, it doesn't mean that airflow isn't running nor installed improperly. You'd have an easier time testing your install with airflow list_dags and you probably must run airflow initdb before most of the commands (and look at the airflow.cfg file).
On trying to install MySql on RHEL7 I came to know that we need to use MariaDB instead.
I tried the installation on my machine which is RHEL6 by adding MariaDB.repo file to /etc/yum.repos.d/ with following lines -
[mariadb]
name = MariaDB-5.5.39
baseurl=https://downloads.mariadb.com/files/MariaDB/mariadb-5.5.39/yum/rhel6-amd64/
# alternative: baseurl=http://archive.mariadb.org/mariadb-5.5.39/yum/rhel6-amd64/
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
https://mariadb.com/kb/en/mariadb/yum/
Thereafter I installed it successfully using following command -
sudo yum install MariaDB-server MariaDB-client
Now on an RHEL7 production box same steps failed with following conflict -
file /etc/my.cnf from install of MariaDB-common-5.5.39-1.el6.x86_64 conflicts with file from package maria-libs-1:5.5.44-2.el7.x86_64
So I understand I am pointing to rhel6 repo on rhel7 and probably some pre-installed files are in conflict.
But when I execute below mysql/mariadb gets installed successfully, what difference does case make here?
sudo yum install mariadb mariadb-server
I am not sure if it is linked to a different repository in background, I dont have access to the instance to dig further in repo settings, can anyone help me out here?
You can clarify that using yum search mariaDB
I executed that an ended up with same result as mariaDB and mariadb
============================= N/S matched: mariaDB =============================
mariadb-bench.x86_64 : MariaDB benchmark scripts and data
mariadb-devel.i686 : Files for development of MariaDB/MySQL applications
mariadb-devel.x86_64 : Files for development of MariaDB/MySQL applications
mariadb-libs.i686 : The shared libraries required for MariaDB/MySQL clients
mariadb-libs.x86_64 : The shared libraries required for MariaDB/MySQL clients
mariadb-server.x86_64 : The MariaDB server and related files
mariadb.x86_64 : A community developed branch of MySQL
mariadb-test.x86_64 : The test suite distributed with MariaD
Name and summary matches only, use "search all" for everything.
When trying to migrate I get the error:
'PDOException' with message could not find driver
I'm on an Ubuntu Linux environment. I've configured "sqlite" and if I run sqlite3 from the command line I get to the sqlite shell, so it is correctly installed, but Laravel does not connect to the database!
If you have already installed the php_pdo, but it is not showing on your phpinfo(), you might need to add extension=sqlite.so to your /etc/php5/conf.d/sqlite.ini file.
pdo_slqite is missing. Should be installable via...
sudo apt-get install php5-sqlite