I am running airflow in containers AWS ECS, 1 scheduler, 2 web servers, and multiple celery workers.
From what I have seen the only thing that is affected when running them in containers is that the web servers are unable to access the workers' port on 8793 to retrieve logs from the workers.
Is that that only thing that is affected when running these in containers?
Yes because you can only map one port from docker container to host instance. I use a similar setup and logs are the only main issue I have faced. There are different ways to solve this though. You use other logging services on the container which push the logs to Cloudwatch or FLuentdb etc..
Related
I'm a newbie at using Airflow. I went through many Airflow tutorials, and I can say that all are about development environments using a docker-compose file or files. I'm facing a problem at work setting up a production environment properly. My goal is to have a cluster composed of 3 EC2 virtual machines. Can anyone share best practices for installing Airflow on that cluster?
I went through many tutorials on the internet.
Airflow has 4 main components:
Webserver: stateless service which expose the UI and the REST API of Airflow
Scheduler: stateless service which processes the dags and runs them and it's the main component
Worker: stateless service which execute the tasks
Metadata: the database of Airflow where the state is stored, and it manages the communication between the 3 other components
And Airflow has 4 main executors:
LocalExecutor: the scheduler runs tasks by itself by spawning a process for each task, and it works in a single host -> not suitable for your need
CeleryExecutor: the most used scheduler, you can create one or multiple scheduler (for HA), and a group of celery workers to run the tasks, you can scale it on different nodes
DaskExecutor: similar to CeleryExecutor but it uses Dask instead of Celery, not much used, and there is no many resources around it
KubernetesExecutor: it runs each task in a K8S pod, and since it's based on Kubernetes, it's very scalable, but it has some drawbacks.
For you use case, I recommend using CeleryExecutor.
If you can use EKS instead of EC2, you can use the helm chart to install and configure the cluster. And if not, you have other options:
run the services directly on the host:
pip install apache-airflow[celery]
# run the webserver
airflow webserver
# run the scheduler
airflow scheduler
# run the worker
airflow celery worker
You can decide how many scheduler, workers and webserver you want to execute, and you can distribute them on the 3 nodes, for ex: node1(1 scheduler, 1 webserver, 1 worker), node2(1 scheduler, 2 workers), node3(1 webserver, 2 workers), and you need a DB, you can use postgres from AWS RDS, or create it on one of the nodes (not recommended).
using docker: same as the first solution, but your run containers instead of running the services directly on the host
using docker swarm: you can connect the 3 nodes to create a swarm cluster, and manage the config from one of the nodes, this gives you some feature which are not provided by the first 2 solutions, and it's similar to K8S. (doc)
For the 3 solutions, you need to create airflow.cfg file contains the configurations and the DB creds, and you should set the exeutor conf to CeleryExecutor.
We are working on switching over to micro services from a monolithic application.
Each microservice is going to be running on Docker through Amazon ECS.
We've decided to use Consul for service discovery. We have 3 servers running on EC2 instances inside the VPC.
My question is as follows:
How/Where do I start the Consul agent for each micro service? Do I run another container on each instance (through Docker-Compose) with Consul inside? Or do I somehow run a Consul agent inside the already existing Docker container for each micro service?
Attached is a rough representation of my situation.
Should the Consul Client (in yellow) be in its own Docker Container or inside the Node.js container?
Consul is another service, and I wouldn't deploy it inside the container of my microservice. In a large-scale scenario, I'd deploy several Consul containers: some would run the agent under Server mode (think of them as Masters), and some would run it under Client mode (think of them as Slaves).
I wouldn't deploy the agents running under client mode as part of my application's containers, because:
Isolating them means that they are stopped individually. Putting them together means that whenever I'd stop my application's container due to a version upgrade or a failure, I'd be needlessly stopping the Consul agent running therein. Same goes the other way around: stopping the Consul agent would stop my running application. This unneeded coupling isn't beneficial.
Isolating them means they can be scaled separately. I may need to scale my microservice and deploy more instances of it. If the container also contains Consul client agents, then scaling my microservice would end up scaling Consul as well. Or the other way around: I may need to scale Consul without scaling my microservice.
Isolating them is easier in terms of Docker container images. I can keep using the official Consul image and upgrade without much hassle. Putting Consul and my microservice together would mean that upgrading Consul requires me to modify the container image by myself.
I am pretty much new to the docker concept and know basics of it.
I just wanted to know how can we build multi tenant application using docker.
Where the containers will use the local hosted database with different schema.With the nginx we can do reverse proxy but how we can achieve it?
because every container will be accessed by localhost:8080 and how we can add upstream and server part.
It will be very helpful if some one explains it to me.
If I understand correctly you want processes in containers to connect to resources on the host.
From you containers perspective in bridge mode (the default), the host's IP is the gateway. Unfortunate the gateway IP address may vary and can only be determinate at runtime.
Here are a few ways to get it:
From the host using docker inspect: docker inspect <container name or ID>. The gateway will be available under NetworkSettings.Networks.Gateway.
From the container you can execute route | awk '/^default/ { print $2 }'
One other possibility is to use --net=host when running your container.
This will run you processes on the same network as your processes on your host. Doing so will make your database accessible from the container on localhost.
Note that using --net=host will not work on Docker for mac/windows.
Running a Postgresql DB on a Docker Swarm containing multiple nodes where the Database can be deployed. Using Docker version 1.12+.
Using a Data container, the Postgresql failover is working nicely. Now I would like to have a Java client connect to the DB and also survive failover. How should the JDBC connections be managed here? Does the connection string change? Or should it be managed through something like an nginx container running elsewhere? Is there an example of this available for study anywhere? Conceptually, I think I get moving this off to another (nginx-like) container, but can't quite grok the details!
In swarm mode, you get service discovery by DNS name for services in the same overlay network, you don't need to add a proxy layer yourself. The swam networking docs go into detail, but in essence:
docker network create -d overlay app-net
docker service create --name app-db --network app-net [etc.]
docker service create --name app-web --network app-net [etc.]
Your database server is available by DNS within the network as app-db, to any service in the same app-net network. So app-db is the server name you use in your JDBC connection string. You can have multiple replicas of the Postgres container, or a single container which moves around at failover - the service will always be available at that address.
But: I would be cautious about failover with your data container. You have a single container with your database state there; even if your state is in a volume, it won't move around the cluster. So if the node with the data fails, your data container will start somwhere else, but the data won't go with it.
I have a web service (webpage) that allows the user to configure the network interfaces of the host (it is basically a webpage used to configure the host NICs). Now we are thinking of moving such service inside a docker container. That means that the sw running inside the container should be able to modify the configuration of the network interface of the host the docker is running on top of.
I tried starting a docker with --network=host and I used the ip command to modify the interfaces configuration, but all I can (obviously?!?) get is permission denied.
This probably make sense as it might be an issue from a security point of view, not to mention you are changing the network configuration seen by other potentially running containers, but I'm wondering if there is any docker configuration/setting that might allow me to perform the task entirely inside the docker container (at my own risk).
By that I mean that I can think at least of a workarond, having a service running on the host (outside the docker container) and have the docker and the service talk to each other with some IPC mecchanics.
This is a solution, but not optimal, as this will brake the docker paradigm of having all your stuff running inside the container. Moreover that would mean that when we upgrade the container with a new version of the software, we might need also to upgrade the module outside the container.
Try running your container in privileged mode to remove the container restrictions:
docker run --net=host --privileged ...
If that solves your issue, you can likely replace the --privileged with --cap-add and various kernel capabilities. The first privilege that comes to mind is NET_ADMIN, which you could try with:
docker run --net=host --cap-add NET_ADMIN ...
See this section of the docker run docs for more details on configuring privileges.