I have started learning/trying Airflow. Part of my research, I have installed airflow in my local through Docker. I referred official page Airflow Install Docker
I am looking for a (standard) process when I can deploy Airflow to Azure.
Can I directly use same docker-compose file for that ?
Any help will be appreciated.
Likely the easiest way is to use AKS (Azure Kubernetes Service) and use the official Helm Chart of Apache Airflow from the Apache Airflow community to deploy Airflow on it:
https://airflow.apache.org/docs/helm-chart/stable/index.html
I reckon, Installing Airflow through Docker-compose.yml on Azure wouldn't be the right approach. It is probably encouraged for local installations only.
The way we have set this up in Azure is like this:
Database: Azure Database for PostgreSQL server
Webserver + Scheduler: Azure App Service on a Linux App Service Plan
Docker: Azure Container Registry
Build the Docker image (either locally or in CICD pipeline)
Push the Docker image to Azure Container Registry
Setup Postgres server and create airflow database
Setup Azure App Service to pull Docker image from ACR (You can also setup continuous integration). Remember to configure Environment variables.
Obviously, as Jarek mentioned, you could also go with setting up AKS.
You can try the preconfigured images of Kubernertes AirFlow2 packaged by Bitnami. And perform the Deploy in AKS
bitnami.airflow
Related
I have previous experience of running Astronomer (Airflow - through Astro CLI ) in my local Docker. But now, I have been provided with AWS VPN. I also Astro cloud UI. Do I need Docker to be installed in AWS VPN ? If so how?
Thanks
Sabarisri
I am using gitlab ci/cd to deploy my app to google app engine. I already have php instance working properly but when i try build wordpress image using docker-compose, nothing happen.
these are my files:
I have a folder "web" with a files ping.php: https://site-dot-standalone-applications.appspot.com/ping.php
So application is running into /web folder.
wordpress should be deployed into /web folder after:
docker-compose up
UPDATE
Just needed use the following gitlab-ci.yaml:
Unfortunately, you cannot (easily) deploy containers to App Engine Flex this way.
At its simplest, App Engine Flex is a service that combines a load-balancer, an auto-scaler and your docker image. Your image when run as a container is expected to provide an HTTP/S endpoint on port 8080.
There are 2 ways that App Engine could support your deployment but it does neither:
It bundles a WordPress app image and a MySQL image into a single "pod" and exposes WordPress' HTTP port on :8080. This isn't what you want because then each WordPress instance has its own MySQL instance.
It separates the WordPress app into one service and the MySQL app into another service. This is closer to what you want as you could then scale the WordPress instances independently of the MySQL instances. However, databases are the definitive stateful app and you don't want to run these as App Engine services.
The second case suggests some alternative approaches for you to consider:
Deploy your WordPress app to App Engine but use Google Cloud SQL service link.
If you don't want to use Cloud SQL, you could run your MySQL database on Compute Engine link.
You may wish to consider Kubernetes Engine. This would permit both the approaches outlined above and there are tools that help you migrate from docker-compose files to Kubernetes configurations link.
Since you're familiar with App Engine, I recommend you consider using option #1 above (Cloud SQL)
I'm new to Azure Container Services(ACS) . I have ASP.NET site I need to deploy from Docker Hub to ACS. IS it any easiest way to deploy from Docker Hub?
I don't know the detailed requirements, but there is the possibility to deploy WebApps (as you said you have ASP.NET site) directly from Docker Hub: https://learn.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image.
Alternatively, you could maybe use Azure Container Registry to register and manage your containers over there...
I have a running dev installation of a dockerized wordpress project( via docker-compose) and was wondering what would be the best approach to deploying it to Azure. Most tutorials assume you are starting from scratch however I already have an installation and was wondering if I just need to integrate the azure-cli to the compose or setup an azure and migrate what was already done.
I have an account but do I invoke a docker VM or a wordpress from azure ??
What about the database?
The easiest path is for you to create a new Azure Container Service, which can host Docker Swarm, and deploy it there. Install the Azure CLI 2.0 on your machine and follow the tutorial to get started.
We occasionally have the need to restart services that are deployed with AWS CodeDeploy. Is it possible to have the CodeDeploy agent do this directly, without having to create a new deployment?
The AWS service you're looking for is AWS Systems Manager. You can run arbitrary commands or scripts on instances with this2. All recent Ubuntu and Amazon Linux instances have AWS SSM agent installed. But if you have an older instance, you'll have to install the SSM agent manually or through your configuration manager.
No, you need to have a deployment to restart. The agent does not take actions on its own. It receives commands from the CodeDeploy service.
Depending on your usecase you can have your application emit CloudWatch event and have that trigger a deployment in the deploymentGroup. Note that it will create a deployment that will deploy to the entire fleet.
To expand on eternaltyro's answer, you could leverage CodeDeploy's CLI tool via SSM to run the same CodeDeploy event hooks that were/are used to start and stop your application.