I'm using Docker Compose to create two containers. One runs an Nginx web server which serves the mydomain.com website, and the second needs to send HTTP requests to the first one (using the mydomain.com domain name).
I don't want to have to check the Nginx container's ip each time I run it and then use docker run --add-host on the second container. My goal is to run docker-compose up and that everything be ready.
I know it's not possible, but what I'm looking for is something in the line of:
# docker-compose.yml
nginx_container:
...
second_container:
extra_hosts:
# This is invalid. extra_hosts only accepts ips.
- "mydomain.com:nginx_container"
You can get a similar result using a configuration like this:
version: "3"
services:
api:
image: node:8.9.3
container_name: foo_api
domainname: api.foo.test
command: 'npm run dev'
links:
- "mongo:mongo.foo.test"
- "redis:redis.foo.test"
volumes:
- .:/app
- /app/node_modules
ports:
- "${PORT}:3000"
- "9229:9229"
depends_on:
- redis
- mongo
networks:
- backend
redis:
image: redis:3
container_name: foo_redis
domainname: redis.foo.test
ports:
- "6379:6379"
networks:
- backend
mongo:
image: mongo:3.6.2
container_name: foo_mongo
domainname: mongo.foo.test
ports:
- "${MONGO_PORT}:27017"
environment:
- MONGO_PORT=${MONGO_PORT}
networks:
- backend
networks:
backend:
driver: bridge
Related
Firstly, I'd like to say that I'm not a server admin. I'm a web programmer tasked with setting up a development server and I have no idea what I'm doing. I may not be doing things according to best practice or the way you might do them. Unfortunately, with Traefik, there are 3 ways to do everything and so 2/3 of the answers that I've come across aren't compatible with my implementation and I can't figure out how to make them work. Furthermore, this isn't my only (or even primary) job duty.
Here's the setup:
Single-server docker environment on a Linode server with Ubuntu 20.04
I have one stack with Traefik, Traefik Hub, Portainer, and WhoAmI configured and working (mostly) correctly. I don't have the DNS challenge working right with Let's Encrypt, but I don't really care about that at this point. I don't really need a wildcard certificate.
I created a mariadb container. We're mostly a WordPress shop and I'd like to have one container for all the databases we work with rather than configuring a database on an environment-by-environment basis.
I created an external bridge network, named "maverick-net" and all of the stacks are connected to it.
I have a self-hosted GitHub runner listening for changes to the "dev" branch of the project. The runner pulls down the latest repo, writes GitHub secrets to a local .env file, runs composer install and then docker-compose up -d. (That's the reason behind the obscenely-long bind mount paths.)
I'm trying to make the code for these WordPress projects reusable as much as possible, so there's a lot of .env variables in the different files. At some point I'll probably move those over to docker secrets, but at this point it's a development server and not as critical.
My issue is that I haven't been able to get a WordPress site up and running, and I keep hitting a "Bad Gateway" error. When I curl the URL from inside the traefik container, I get... wait for it... "Bad Gateway."
Clearly there's something I'm missing, but I've been slamming my head against a brick wall for weeks trying different approaches to get this running and I need help. There has to be something I'm not getting about docker networks in general because my wp-cli container never has been able to connect to the database, regardless of whether I start it in the same stack or if i try to connect to it on the maverick-net network.
My traefik stack (side note, I'd really like to split these command entries into static and dynamic config files, but that's a task for another day):
version: "3.9"
secrets:
linode_token:
file: "../secrets/linode_token.secret"
services:
traefik:
container_name: traefik
image: "traefik:latest"
command:
- --log.level=DEBUG
- --log.filePath=./traefik.log
- --accessLog=true
- --accessLog.filePath=./access.log
- --accessLog.bufferingSize=100
- --accessLog.filters.statusCodes=400-499
- --api
- --api.dashboard=true
- --api.insecure=false
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --providers.docker
- --providers.docker.watch=true
- --providers.docker.exposedbydefault=false
- --certificatesresolvers.leresolver.acme.dnsChallenge=true
- --certificatesresolvers.leresolver.acme.dnsChallenge.provider=linodev4
- --certificatesresolvers.leresolver.acme.httpchallenge=true
- --certificatesresolvers.leresolver.acme.httpchallenge.entrypoint=web
- --certificatesresolvers.leresolver.acme.email=xxxxxxxxxxx#xxxxxxxxx.xxx
- --certificatesresolvers.leresolver.acme.storage=./acme.json
#- --certificatesresolvers.leresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
- --certificatesresolvers.leresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
- --experimental.hub=true
- --hub.tls.insecure=true
- --metrics.prometheus.addrouterslabels=true
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ~/certs-data/acme.json:/data/letsencrypt/acme.json
- ./static.yml:/static.yml:ro
- ./configs:/configs
- ~/certs-data/:/data/letsencrypt/
secrets:
- "linode_token"
environment:
TZ: America/Chicago
LINODE_TOKEN_FILE: "/run/secrets/linode_token"
labels:
- "traefik.enable=true"
- "traefik.docker.network=maverick-net"
- "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
- "traefik.http.routers.http-catchall.entrypoints=web"
- "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
- "traefik.http.routers.traefik.tls.certresolver=leresolver"
- "traefik.http.routers.traefik.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.traefik.entrypoints=websecure"
- "traefik.http.routers.traefik.service=api#internal"
- "traefik.http.routers.traefik.middlewares=traefik-auth"
- "traefik.http.middlewares.traefik-auth.basicauth.users=XXXX:$$apr1$$XXXXX$$XXXXXXXXXXXXXXX"
- "traefik.http.routers.api.entrypoints=websecure"
networks:
- maverick-net
hub-agent:
image: ghcr.io/traefik/hub-agent-traefik:experimental
pull_policy: always
container_name: hub-agent
restart: on-failure
command:
- run
- --hub.token=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX
- --auth-server.advertise-url=http://hub-agent
- --traefik.host=traefik
- --traefik.tls.insecure=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on:
- traefik
networks:
- maverick-net
portainer:
image: portainer/portainer-ce:latest
command: -H unix:///var/run/docker.sock
container_name: portainer
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- portainer_data:/data
labels:
# Frontend
- "traefik.enable=true"
- "traefik.http.routers.frontend.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.frontend.entrypoints=websecure"
- "traefik.http.services.frontend.loadbalancer.server.port=9000"
- "traefik.http.routers.frontend.service=frontend"
- "traefik.http.routers.frontend.tls.certresolver=leresolver"
networks:
- maverick-net
whoami:
image: "traefik/whoami"
container_name: "whoami"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls.certresolver=leresolver"
networks:
- maverick-net
volumes:
portainer_data:
networks:
maverick-net:
external: true
My mariadb stack:
version: "3"
networks:
# enable connection with Traefik
maverick-net:
external: true
services:
mariadb:
container_name: mariadb
image: mariadb:10.7
restart: always
volumes:
- "/home/xxxxxxxxxx/docker/mariadb/data:/var/lib/mysql"
expose:
- "3306"
env_file: .env
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PWD}
MYSQL_USER: ${ADMIN_DB_USER}
MYSQL_PASSWORD: ${ADMIN_DB_PWD}
networks:
- maverick-net
And finally my WordPress stack:
version: '3.8'
networks:
maverick-net:
external: true
# volumes:
# db_data:
services:
# mariadb:
# container_name: ${WORDPRESS_DB_NAME}-db
# image: mariadb:10.7
# restart: always
# volumes:
# - "db_data:/var/lib/mysql"
# env_file: .env
# environment:
# MYSQL_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
# MYSQL_USER: ${ADMIN_DB_USER}
# MYSQL_PASSWORD: ${ADMIN_DB_PWD}
wordpress:
container_name: ${WORDPRESS_DB_NAME}-wp
image: wordpress:6.0.2-fpm
volumes:
- type: bind
source: ${PROJECT_ROOT}/${WORDPRESS_DB_NAME}/${PROJECT_NAME}/${PROJECT_NAME}/wp
target: /var/www/html
restart: always
env_file: .env
environment:
WORDPRESS_DB_HOST: mariadb
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
WORDPRESS_DATABASE_USER: ${WORDPRESS_DB_USER}
WORDPRESS_DATABASE_PASSWORD: ${WORDPRESS_DB_PASSWORD}
WORDPRESS_DATABASE_NAME: ${WORDPRESS_DB_NAME}
labels:
# The labels are useful for Traefik only
- "traefik.enable=true"
- "traefik.docker.network=maverick-net"
# Get the routes from https
- "traefik.http.routers.${WORDPRESS_DB_NAME}.rule=Host(`${DEV_URL}`)"
- "traefik.http.routers.${WORDPRESS_DB_NAME}.entrypoints=websecure"
- "traefik.http.routers.${WORDPRESS_DB_NAME}.tls.certresolver=leresolver"
networks:
- maverick-net
wordpress-cli:
container_name: ${WORDPRESS_DB_NAME}-cli
image: wordpress:cli
volumes:
- type: bind
source: ${PROJECT_ROOT}/${WORDPRESS_DB_NAME}/${PROJECT_NAME}/${PROJECT_NAME}/wp
target: /var/www/html
env_file: .env
environment:
WORDPRESS_DB_HOST: mariadb
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
WORDPRESS_DATABASE_USER: ${WORDPRESS_DB_USER}
WORDPRESS_DATABASE_PASSWORD: ${WORDPRESS_DB_PASSWORD}
WORDPRESS_DATABASE_NAME: ${WORDPRESS_DB_NAME}
networks:
- maverick-net
depends_on:
- wordpress
As far as I know, you can connect to containers in the same network by using their service name.
So for example you are trying to curl to the Wordpress container from the Traefik Container.
curl 'http://wordpress/'
Should work.
In another project I use an nginx container with php-fpm.
I need to send my curl requests to the nginx container, because the php-fpm container does not handle server requests directly:
// does not work
curl 'http://php-debug/index.html'
// result
curl: (7) Failed to connect to php-debug port 80: Connection refused
// https also does not work
curl 'https://php-fpm/index.html'
// result
curl: (7) Failed to connect to php-fpm port 443: Connection refused
// This does work
curl 'http://nginx/index.html'
// result
<HTML...
For some reason https: curl requests fail, but on http I get the correct result so for local development I think it's ok.
If you are interested in a more managed solution, you can check out warden.dev. It contains a template for Wordpress too (which I use succesfully for local development). I have been using this exclusively. If you have questions how to setup WP CLI on this solution, feel free to contact me.
It comes with portainer, traefik, ssl and dns and mailhog.
Configuration is pretty straight forward, I can set up a new project within an hour and connect to the database and containers in my IDE.
https://docs.warden.dev/environments/types.html#wordpress
I have the following docker-compose.yml, but need to model a public/private network split where the Redis instance must only be accessible to localhost.
version: "2.2" # for compatibility, I can upgrade if needed
services:
nginx:
image: nginx
ports:
- 8080:80
redis:
image: redis
ports:
- 6379:6379
This seems straightforward if I needed to restrict it to being accessible only within the docker network. Consider:
version: "2.2"
services:
nginx:
image: nginx
ports:
- 8080:80
networks:
- frontend
- backend
redis:
image: redis
ports:
- 6379:6379
networks:
- backend
networks:
frontend: {}
backend:
internal: true
However our local web developers need to be able to access that Redis instance from their host machine (outside of the docker network) when they build, run, and debug locally.
Just bind the service port of redis to localhost(127.0.0.1).
Try follows...
...
redis:
image: redis
ports:
- 127.0.0.1:6379:6379
...
Run Redis Web UI called redis-commander.
Use env vars to point to the running redis.
Expose this new container & access it instead of exposing Redis container.
services:
redis:
# Do comment ports! no need to expose it
# ports:
# - 6379:6379
// ....
redis-commander:
image: rediscommander/redis-commander:latest
restart: unless-stopped
environment:
REDIS_HOST: redis:6379 # <-- 🔴 Here you point to your redis
# REDIS_PASSWORD # <- in case redis is protected with password
ports:
- "8081:8081"
Let your developers go to http://localhost:8081 and enjoy.
Find more details about that image
Description
I am trying to build an equal configuration in my local docker-environment like on our production system. After spending some time investigating and rebuilding the docker container setup, still can't get it to work and Graylog is not receiving any data.
Overview and interim results
web, php and db container are in use for the symfony based application
symfony runs properly on localhost in php-container and generates logfiles
symfony-logfiles are located here: /var/www/html/var/logs/*.log
symfony-logfiles format is json / gelf
all other containers are also up and running when starting the complete composition
filebeat configuration is based on first link below
filebeat.yml seems to retrieve any logfile found in any container
filebeat configured to transfer data directly to elasticsearch
elasticsearch persists data in mongodb
all graylog related data in persisted in named volumes in docker
additionally I am working with docker-sync on a Mac
The docker-compose.yml is based on the following resources:
https://github.com/jochenchrist/docker-logging-elasticsearch
http://docs.graylog.org/en/2.4/pages/installation/docker.html?highlight=docker
https://www.elastic.co/guide/en/beats/filebeat/6.3/running-on-docker.html
https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-reference-yml.html
config.yml
# Monolog Configuration
monolog:
channels: [graylog]
handlers:
graylog:
type: stream
formatter: line_formatter
path: "%kernel.logs_dir%/graylog.log"
channels: [graylog]
docker-compose.yml
version: "3"
services:
web:
image: nginx
ports:
- "80:80"
- "443:443"
links:
- php
volumes:
- ./docker-config/nginx.conf:/etc/nginx/conf.d/default.conf
- project-app-sync:/var/www/html
- ./docker-config/localhost.crt:/etc/nginx/ssl/localhost.crt
- ./docker-config/localhost.key:/etc/nginx/ssl/localhost.key
php:
build:
context: .
dockerfile: ./docker-config/Dockerfile-php
links:
- graylog
volumes:
- project-app-sync:/var/www/html
- ./docker-config/php.ini:/usr/local/etc/php/php.ini
- ./docker-config/www.conf:/usr/local/etc/php-fpm.d/www.conf
db:
image: mysql
ports:
- "3306:3306"
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
- MYSQL_DATABASE=project
- MYSQL_USER=project
- MYSQL_PASSWORD=password
volumes:
- ./docker-config/mysql.cnf:/etc/mysql/conf.d/mysql.cnf
- project-mysql-sync:/var/lib/mysql
# Graylog / Filebeat
filebeat:
build: ./docker-config/filebeat
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
networks:
- graylog-network
depends_on:
- graylog-elasticsearch
graylog:
image: graylog/graylog:2.4
volumes:
- graylog-journal:/usr/share/graylog/data/journal
networks:
- graylog-network
environment:
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
links:
- graylog-mongo:mongo
- graylog-elasticsearch:elasticsearch
depends_on:
- graylog-mongo
- graylog-elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
graylog-mongo:
image: mongo:3
volumes:
- graylog-mongo-data:/data/db
networks:
- graylog-network
graylog-elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.10
ports:
- "9200:9200"
volumes:
- graylog-elasticsearch-data:/usr/share/elasticsearch/data
networks:
- graylog-network
environment:
- cluster.name=graylog
- "discovery.zen.minimum_master_nodes=1"
- "discovery.type=single-node"
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
# Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
- xpack.security.enabled=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
project-app-sync:
external: true
project-mysql-sync: ~
graylog-mongo-data:
driver: local
graylog-elasticsearch-data:
driver: local
graylog-journal:
driver: local
networks:
graylog-network: ~
Dockerfile of filebeat container
FROM docker.elastic.co/beats/filebeat:6.3.1
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
# must run as root to access /var/lib/docker and /var/run/docker.sock
USER root
RUN chown root /usr/share/filebeat/filebeat.yml
# dont run with -e, to disable output to stderr
CMD [""]
filebeat.yml
filebeat.prospectors:
- type: docker
paths:
- '/var/lib/docker/containers/*/*.log'
# path to symfony based logs
- '/var/www/html/var/logs/*.log'
containers.ids: '*'
processors:
- decode_json_fields:
fields: ["host","application","short_message"]
target: ""
overwrite_keys: true
- add_docker_metadata: ~
output.elasticsearch:
# transfer data to elasticsearch container?
hosts: ["localhost:9200"]
logging.to_files: true
logging.to_syslog: false
Graylog backend
After setting up this docker composition I started the Graylog web-view and set up a collector and input as described here:
http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#step-by-step-guide
Maybe I have totally misunderstood how this could work. I am not totally sure if Beats from Elastic is the same as the filebeats container and if the sidecar collector is something extra I forgot to add. Maybe I misconfigured the collector and input in graylog?!
I would be thankful to any help or working example according to my problem ...
Graylog seems to be running on http://127.0.0.1:9000/api which is in the container. You might want to run it as http://graylog:9000/api or as http://0.0.0.0:9000/api
Accessing the other images from within any of the other images will have be done with the same name as the service name, as defined in the docker-compose.yml files. The url to the graylog-elasticsearch would be something like: http://graylog-elasticsearch/.... if you would post to localhost it would stay inside its own image.
Hope this will help you along in finding the solution.
I have two Docker containers: node-a, node-b. One of them (node-b) should send http request to other (node-a). I'm starting them with Docker Compose. When I'm trying to up them with Compose I face an error:
Get http://node-a:9098: dial tcp 172.18.0.3:9098: getsockopt: connection refused
EXPOSE is declared in Docker file of a-node:
EXPOSE 9098
docker-compose.yml:
version: '3'
services:
node-a:
image: a
ports:
- 9098:9098
volumes:
- ./:/a-src
depends_on:
- redis
node-b:
image: b
volumes:
- ./:/b-src
depends_on:
- node-a
Forwarding is enabled. I believe a server starts because it works well without Docker.
Where I should pay attention? What could cause a problem?
EDIT:
I've tried to add links but it had no effect:
node-b:
image: b
volumes:
- ./:/b-src
links:
- node-a
depends_on:
- node-a
Also links seemed to be deprecated and does the same thing as depends_on in 2+ version of docker-compose.yml:
docker-compose execute V2 files, it will automatically build a network between all of the containers defined in the file, and every container will be immediately able to refer to the others just using the names defined in the docker-compose.yml file.
Link a container to the service using links. (docker-compose documentation on links).
Example:
node-b:
image: b
volumes:
- ./:/b-src
depends_on:
- node-a
links:
- node-a
I am running two docker containers, using the following docker-composer script:
db:
image: cofoundry/mysql
environment:
- MYSQL_USER=wordpress
- MYSQL_PASS=wordpress
- MYSQL_DB=wordpress
ports:
- "3306:3306"
privileged: true
web:
image: cofoundry/nginx-phpfpm
volumes:
- ../bedrock:/app
environment:
- DOCROOT=/app/web
links:
- db
ports:
- "80:80"
privileged: true
But I'm not sure how can I go about running multiple nginx-phpfpm web containers with different sites, like web_1, web_2 etc. ?
Or, alternatively, how to propagate custom nginx.conf while running docker-composer up, so I can configure virtualhosts there.
Thanks!
So the way to do it (as I mentioend in my comment) is to use something like autodock:
Example: (based on your docker-compose.yml):
db:
image: cofoundry/mysql
environment:
- MYSQL_USER=wordpress
- MYSQL_PASS=wordpress
- MYSQL_DB=wordpress
ports:
- "3306:3306"
privileged: true
web:
image: cofoundry/nginx-phpfpm
volumes:
- ../bedrock:/app
environment:
- DOCROOT=/app/web
- VIRTUALHOST=myapp.example.com
links:
- db
privileged: true
autodock:
image: prologic/autodock
ports:
- "1338:1338/udp"
- "1338:1338/tcp"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
autodockhipache:
image: prologic/autodock-hipache
links:
- autodock
- hipache:redis
hipache:
image: hipache
ports:
- 80:80
- 443:443
The only requirement is that you setup a wildcard DNS entry for *.example.com to point to your "Docker Host". e.g:
*.example.com A 1.1.1.1
You can read more about how this all works in the article:
A Docker-based mini PaaS
This will scale and work for as many websites as you want to run on your "Docker Host".
Edit: The nice thing about this setup is that it's simple, it's automated and it only requires you add a single extra environment variable to the container you want to expose to the web with only one initial setup requirements (wildcard dns entry).