Please tell me, I install the airflow locally through the docker compose on window 10. The installation is already in its fourth hour. Is this how it should be?
Airflow installation through docker compose should not be taking 4 hours to complete. At most, the installation can take up to half an hour, perhaps you should try start over your installation process.
Related
There seems to be no proper documentation about upgrading airflow. The Upgrading Airflow to a newer version page only talks about upgrading the database. So what is the proper way of upgrading airflow?
Is it just upgrading the python packages to the newest versions? Or should I use the same venv and install the newer airflow version completely from scratch? Or is it something else altogether?
I'm guessing doing the database upgrade would be the final step followed by one of these steps.
I was also struggling with upgrading airflow for minor versions and didn't feel like I found a good answer in the docs. I think I have the right approach after looking back at how I installed airflow in the first place.
If you followed the guide to run airflow locally you'll want to change the value for AIRFLOW_VERSION in the commands to your desired version.
If you followed the guide to run airflow on docker, then you'll want to fetch the latest docker-compose.yaml. The command on the site always has the latest version. Then re-run docker compose up.
You can confirm you have the right version by running airflow version. I run airflow via docker so the docker steps work for me, I imagine the local steps should be about the same.
Adding to Vivian's answer -
I had installed airflow from PyPi and was upgrading from 2.2.4 to 2.3.0.
To upgrade airflow,
I installed the new version of airflow in the same virtual environment as 2.2.4 (using this).
Upgraded the database using airflow db upgrade. More details here.
You might have to manually upgrade providers using pip install packagename -U
After this, when I started Airflow, I got an error related to some missing conf. Airflow wanted the newest version of airflow.cfg, but I had the older version. To fix this,
Renamed my airflow.cfg to airflowbackup.cfg. This is done so that airflow will make a new airflow.cfg on start up when it sees that there is no config file.
Compared airflowbackup.cfg with the 2.2.4 config to find out all the fields I had changed.
Manually made those same changes in the newly made airflow.cfg
So, I'm trying to install Airflow on my laptop and I was able to do that using WSL but I can't seem to open it.
My webserver is running but I cant open it on localhost:8080
The system is taking eons to run airflow scheduler
I am trying to get Airflow running on a local Docker Desktop for Mac installation using the Quick Start instructions for a Docker installation at https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html. The install seems to go correctly, but when I start up the services using docker-compose up, all the services launch except for the webserver. That service starts, but then fails and restarts, over and over. The log shows this error:
EDIT: I found the issue in case someone else runs into it. The default configuration of Docker desktop for Mac doesn't allocate enough memory to run webserver. I bumped up the memory allocation in Docker, restarted everything, and all was good.
increasing the memory configuration of the docker worked for me
Edit:
Docs for docker configurations on mac: https://docs.docker.com/docker-for-mac/
I am busy setting up a dockerized environment to develop PHP for Magento.
The image I am using is the following: alexcheng/magento2.
The git repository for this does contain an install script.
When I run "docker-compose up -d" everything works fine but I have to install Magento afresh each time the container goes down.
Any advice for how to deal with this? I am a relative newb at using docker but I can't imagine that you would have to reinstall it each time.
Note, I don't think this has to do with data persistence as a volume has been provisioned. When I include a line in the Dockerfile "RUN install-magento" I get the following error when building:
/usr/bin/env: ‘bash\r’: No such file or directory
The command '/bin/sh -c install-magento' returned a non-zero code: 127
Any guidance would be appreciated. Thank you.
I'm newbie about Docker as you. I suggest that you learn docker before wildly uses docker-compose tool. When we need data persistence the way is to use docker volumes.
Check if your docker-compose.yml have a volume section and if your user have permission on that path.
My OS is Mac OS. I followed airflow official installation guide to install. But when I test: airflow test tutorial print_date 2015-06-01 from airflow testing it doesn't print any output. The result is here.. I wonder did I install it successfully? I've run other commands on the official airflow testing page. They report no error.
So far I see only WARNING output, it doesn't mean that airflow isn't running nor installed improperly. You'd have an easier time testing your install with airflow list_dags and you probably must run airflow initdb before most of the commands (and look at the airflow.cfg file).