Trying to add SFTP connection to Apache Airflow run on the Composer.
Here is some packages' listing;
package-name
version
apache-airflow
2.2.5+composer
apache-airflow-providers-ftp
3.2.0
apache-airflow-providers-google
2022.10.17+composer
apache-airflow-providers-sftp
4.1.0
apache-airflow-providers-ssh
3.2.0
paramiko
2.12.0
sshtunnel
0.4.0
This is the message I get when hitting the Test button:
Any help would be appreciated.
This is an expected behavior on Composer 1 because in Composer 1 if DAG serialization is on, the Airflow webserver is using a base image that doesn’t include customer installed pypi packages.
Therefore, this functionality doesn’t work properly.
You may upgrade to Composer 2 so that everything will work properly.
However, if you want to stay on Composer 1, you may try changing the conn_type from ‘sftp’ to ‘ftp’.
Related
There seems to be no proper documentation about upgrading airflow. The Upgrading Airflow to a newer version page only talks about upgrading the database. So what is the proper way of upgrading airflow?
Is it just upgrading the python packages to the newest versions? Or should I use the same venv and install the newer airflow version completely from scratch? Or is it something else altogether?
I'm guessing doing the database upgrade would be the final step followed by one of these steps.
I was also struggling with upgrading airflow for minor versions and didn't feel like I found a good answer in the docs. I think I have the right approach after looking back at how I installed airflow in the first place.
If you followed the guide to run airflow locally you'll want to change the value for AIRFLOW_VERSION in the commands to your desired version.
If you followed the guide to run airflow on docker, then you'll want to fetch the latest docker-compose.yaml. The command on the site always has the latest version. Then re-run docker compose up.
You can confirm you have the right version by running airflow version. I run airflow via docker so the docker steps work for me, I imagine the local steps should be about the same.
Adding to Vivian's answer -
I had installed airflow from PyPi and was upgrading from 2.2.4 to 2.3.0.
To upgrade airflow,
I installed the new version of airflow in the same virtual environment as 2.2.4 (using this).
Upgraded the database using airflow db upgrade. More details here.
You might have to manually upgrade providers using pip install packagename -U
After this, when I started Airflow, I got an error related to some missing conf. Airflow wanted the newest version of airflow.cfg, but I had the older version. To fix this,
Renamed my airflow.cfg to airflowbackup.cfg. This is done so that airflow will make a new airflow.cfg on start up when it sees that there is no config file.
Compared airflowbackup.cfg with the 2.2.4 config to find out all the fields I had changed.
Manually made those same changes in the newly made airflow.cfg
Just got this error while working with Next-auth
Server Error
Error: Package subpath './providers/google' is not defined by "exports" in C:\Users...\node_modules\next-auth\package.json
Any help?
I think this comes from your next-auth version.
Make sure you're not working with mixed versions' features.
We now have two main versions: Version 3 and Version 4 (beta version).
By Reading the Docs, You can figure it out.
Check this Thread and use the Link to know more about this version issue
You can solve the problem by upgrading the Version 👉 Upgrade your Next-auth for some compatibilities issues
yes, I have the same issue as this, don't panic.
I solved just trying to install and reinstall node_module.
case 1 remove node_module and install again use npm (install success install not information error)
case 2 remove node_module again and install node_module use yarn ( I getting some information error for requiring version nodejs version), so in this case you need to upgrade or downgrade nodejs to another version.
okay that it, I hope this help you.
I had the same error, and in my particular case are be because i been running a old version of node.
first check your version of node
node -v
after upgrate with your package manager of your preference (brew, apt-get, etc)
brew upgrade node
and finally switch your version with a version manager tool of node
npm i -g n
n latest
for me updating the node version fixed the above issue
Try this:
import GoogleProvider from "next-auth/providers/google";
I can see the following in my pip list but when I try to add a Snowflake connection via the GUI, Snowflake is not an option from the dropdown.
apache-airflow-providers-snowflake 2.1.0
snowflake-connector-python 2.5.1
snowflake-sqlalchemy 1.2.3
Am I missing something?
I have had this issue with MWAA recently.
I find that if I select AWS in the drop down and provide the correct snowflake host name etc it works though.
I run into the same issue using the official helm chart 1.3.0.
But finally I was able to make Snowflake connection visible by doing the following steps:
I uninstalled the apache-airflow-providers-google. Not sure whether this is important, but I like to mention it here. I did this, because I got some warnings.
Because with SQLAlchemy 1.4 some breaking changes were introduced, I made sure that version 1.3.24 gets installed. Based on that I choosed the fitting version for snowflake stuff.
So this is my requirements.txt for my custom Airflow container:
apache-airflow-providers-snowflake==2.3.0
pyarrow==5.0.0
snowflake-connector-python==2.5.1
snowflake-sqlalchemy==1.2.5
SQLAlchemy==1.3.24
This is my Dockerfile:
FROM apache/airflow:2.2.1-python3.8
## adding missing python packages
USER airflow
COPY requirements.txt .
RUN pip uninstall apache-airflow-providers-google -y \
&& pip install -r requirements.txt
I had the same issue where my pip freeze showed the apache-airflow-providers-snowflake yet I did not have the provider in the UI. I had to add the line apache-airflow-providers-snowflake to my requirements.txt file and then restart. Then I was able to see the Snowflake provider and connector in the UI.
I'm trying to host a meteor app that uses an old version of meteor.
Every time i try to start the app it will get somewhat through the process of installing the tool, and then i see a message such as:
Killednloading meteor-tool#1.1.3... -
(note how killed somehow overwrites the downloading part of the command line)
Is there a reliable way to install the meteor tool at a specific version?
EDIT:
The Meteor team added a release parameter to their download endpoint. Now you can simply specify the desired version:
curl "https://install.meteor.com/?release=1.3.3.1" | sh
For Windows, a version parameter exists for the choco installer:
choco install meteor --version 1.3.3.1
Original solution
You can use sed for that. Insert it in the middle of curl and sh:
curl https://install.meteor.com/ | sed 's/1.4/1.3.3.1/' | sh
That will replace the release 1.4 (current version) to 1.3.3.1
When you create a meteor app you can specify a release:
meteor create test --release x.y.z
And when you update a meteor app you can do the same:
meteor update --release x.y.z
#Jorge Issa's answer is good if you are installing Meteor from scratch, on a system that never had Meteor installed, however it's subject to change since versions change all the time, so you need to adapt the sed line.
If you have any version of Meteor already installed, as Michel Floyd mentioned, you can always create a project with a specific version by adding the --release flag.
meteor update --release xxxx works fine with you're actually upgrading, but downgrading is a different story.
My recommendation when it comes to upgrading and eventually downgrading, is to use version control (git).
Attempt upgrade and if all is fine, you're in good shape, if not and you want to downgrade, simply clear the file changes in your version control system and use meteor reset to clean your project and rebuild with the previous version.
!Note! meteor reset clears the local mongo database too, so be sure to back that up first if you're going to do that (check mongodump and mongorestore for that)
finally, if you're looking to clean up the clutter from the .meteor folder, you can delete the folder and then run meteor reset in your project: the meteor executable will detect you don't have the needed packages will re-download the packages for the version needed by your project. (This takes a while and if you have many project, can be cumbersome as you need to do this in each project, but if like me you are looking to clear some space, this works fine.)
Try:
meteor update --release x.y.z
Try
choco install meteor --x86 --params="'/RELEASE:1.5.4.4'"
Meteor example app, microscope has version 0.9.4, when I try use meteor run command to launch application .
There have a print:
Sorry, this project uses Meteor METEOR#0.9.4, which is not installed and
could not be downloaded. Please check to make sure that you are online.
I am behind a proxy in corporate.
When I install the Meteor, only latest 1.0.0 can be installed.
How can I do?
Upgrade the app to 1.0.0, and how?
Downgrade the meteor to 0.9.4, and how?
I tried use meteor update --release 0.9.4, not work or using meteor update, other error pop up:
/home/hunter/.meteor/packages/meteor-tool/.1.0.35.hgbesu++os.linux.x86_32+web.browser+web.cordova/meteor-tool-os.linux.x86_32/dev_bundle/lib/node_modules/fibers/future.js:206
throw(ex);
..
Error: tunneling socket could not be established, cause=socket hang up
at Object.Future.wait (/home/hunter/.meteor/packages/meteor-tool/.1.0.35.hgbesu++os.linux.x86_32+web.browser+web.cordova/meteor-tool-os.linux.x86_32/dev_bundle/lib/node_modules/fibers/future.js:326:15)
at _.extend._createSocket (packages/ddp/stream_client_nodejs.js:265)
at _.extend._launchConnection (packages/ddp/stream_client_nodejs.js:142)
at new LivedataTest.ClientStream (packages/ddp/stream_client_nodejs.js:28)
at new Connection (packages/ddp/livedata_connection.js:52)
at Object.DDP.connect (packages/ddp/livedata_connection.js:1581)
This should proxy issue.
No document there after search.
I was able to successfully update a project from 0.9.3.1 to 1.1 with:
meteor update
Using meteor run like this:
meteor run --release
…won’t upgrade your meteor installation, but will download the --release and run the current app with it.
I had similar issue after updating meteor to 1.4.1. I happened to fix it with this help Github Issue
Reset your app
meteor reset
Install npm with meteor again
meteor npm install
Rebuild npm (Optional)
meteor npm rebuild
This is a bit late and slightly different: I have a similar problem when I upgraded from 1.0.3.1 to 1.0.3.2. After reading the answers here, and here, I tried the following which worked for me:
meteor update --release <new version>
replacing <new version> with the latest version you're running. This updated my project, and subsequent runs using meteor works as expected.
I was to update by change the file .meteor/release to meteor 1.0. I had packages issues, I also noticed that iron router inner actions changed (I fixed this by fixing the files load order) and on Meteor 1.0 you’ve a package for everything.
Try to add the package meteor-platform, it’s a meta package that contains the most used packages.
I also had issue with some atmosphere packages, it’s a tough job to upgrade it.