I followed this [tutorial][1] on getting Airflow v2 set up. This set up come with an example DAG unfortunately they are not visible on my webserver UI dags page.
I have tried to install again but did not help, I have used astro start/ astro stop also without any success
My Dockerfile has only one line FROM quay.io/astronomer/ap-airflow:latest-onbuild
I am a beginner and trying to learn more.
[1]: https://www.astronomer.io/guides/get-started-airflow-2
Related
Learning apache airflow and I'm trying to create a new connection type the correct way, but it still doesn't show up. I am working within a virtual environment on WSL2 through VS Code, and my terminal looks like this (sandbox) nick#GameCube:~$
This is what my pip list within the venv shows
But these are my only conn types within the airflow UI (I am running airflow webserver within the venv too)
You can see that all the other installed conn types show up, but http doesn't for some reason. Why does this happen? I have tried restarting vscode, killing the webserver process and restarting it, etc but it still doesn't show!
The apache-airflow-providers-http==4.0.0 is not compatible with apache-airflow==2.1.0
To solve your issue you need to downgrade http provider to a version compatible with Airflow 2.1.0
Noting that Http provider is installed by default with Airflow if you need to bump its version make sure you set it to compatible version.
In larger context I suggest to install Airflow from constraints as suggested in the docs. This will prevent case of unstable environment as Airflow brings many dependencies and it guarantees a working environment of Airflow. You can upgrade providers to newer versions but you will need to take a closer look on each upgrade. You can read about it in this doc: Installing/upgrading/downgrading providers separately from Airflow core.
I'm customizing some Airflow pages/views as well as adding some of my own. Running in a Docker container, it takes several minutes from the time I apply my changes to either my templates or to Airflow views.py to when refreshing the web ui will reflect the updates. Or I simply have to restart the webserver.
I've tried setting FLASK_ENV=development as well as setting flask_app.debug=True in app.py but I'm still seeing a relatively long lag for changes to take affect. Trying to see if anyone knows of a faster way to trigger the updates, to help speed up my development and debugging. Thanks.
I started reading about how we could run Spark batch jobs using Airflow.
I have tried using SparkSubmitOperator in local and it works fine. However, I need a recommendation, if we could use it in cluster mode.
The only problem I see when using in cluster mode is that, here the application status not being able to be tracked,ref shared in the link below:
https://albertusk95.github.io/posts/2019/12/airflow-tracks-spark-driver-status-cluster-deployment/
Please suggest if anyone has tried using this operator and works well in cluster mode, or if there is any issue using it.
I am following the tutorial in the Airflow docs. When I visit the UI I don't see the toggle to turn on and off (or pause?) the DAGs
I tried to click the trigger DAG button on the right but I guess this just manually runs it once ignoring the scheduler. (A side question, it just says it's running now, it isn't finishing... is it waiting for something?)
So, did I have to do something in order to schedule the DAG first and is that why I'm not seeing a pause button, because it isn't scheduled? that would surprise me because surely I should be able to schedule it from the UI?
Lastly, what are all those other example DAGs and how can I hide them?
Seems to me that some part of your Airflow setup is broken.
Either the scheduler is not working or the files are not deployed.
My suggestion is to check this question as well: Airflow 1.9.0 is queuing but not launching tasks
I tried to install Discourse on DigitalOcean following their tutorial:
https://github.com/discourse/discourse/blob/master/docs/INSTALL-cloud.md
Everything goes fine until one of the lasts points: Bootstrap Discourse.
I run the command ./launcher bootstrap app it take around 5 minutes and then it gets freezed at this point (please see image):
http://postimg.org/image/gjzs7cvpn/
Can anybody help me please?
Thanks a lot
As the document, you should run ./discourse-setup . The configuration need about 2-8 minutes. Just wait.
if you have discourse already installed and you are adding plugins or upgrading to new settings then run ./launcher rebuild app but if installation is first time then remove docker and install new version again