Host-local networks in Docker Compose - networking

I have the following docker-compose.yml, but need to model a public/private network split where the Redis instance must only be accessible to localhost.
version: "2.2" # for compatibility, I can upgrade if needed
services:
nginx:
image: nginx
ports:
- 8080:80
redis:
image: redis
ports:
- 6379:6379
This seems straightforward if I needed to restrict it to being accessible only within the docker network. Consider:
version: "2.2"
services:
nginx:
image: nginx
ports:
- 8080:80
networks:
- frontend
- backend
redis:
image: redis
ports:
- 6379:6379
networks:
- backend
networks:
frontend: {}
backend:
internal: true
However our local web developers need to be able to access that Redis instance from their host machine (outside of the docker network) when they build, run, and debug locally.

Just bind the service port of redis to localhost(127.0.0.1).
Try follows...
...
redis:
image: redis
ports:
- 127.0.0.1:6379:6379
...

Run Redis Web UI called redis-commander.
Use env vars to point to the running redis.
Expose this new container & access it instead of exposing Redis container.
services:
redis:
# Do comment ports! no need to expose it
# ports:
# - 6379:6379
// ....
redis-commander:
image: rediscommander/redis-commander:latest
restart: unless-stopped
environment:
REDIS_HOST: redis:6379 # <-- 🔴 Here you point to your redis
# REDIS_PASSWORD # <- in case redis is protected with password
ports:
- "8081:8081"
Let your developers go to http://localhost:8081 and enjoy.
Find more details about that image

Related

Bad Gateway for WordPress containers behind Traefik reverse proxy in docker-compose

Firstly, I'd like to say that I'm not a server admin. I'm a web programmer tasked with setting up a development server and I have no idea what I'm doing. I may not be doing things according to best practice or the way you might do them. Unfortunately, with Traefik, there are 3 ways to do everything and so 2/3 of the answers that I've come across aren't compatible with my implementation and I can't figure out how to make them work. Furthermore, this isn't my only (or even primary) job duty.
Here's the setup:
Single-server docker environment on a Linode server with Ubuntu 20.04
I have one stack with Traefik, Traefik Hub, Portainer, and WhoAmI configured and working (mostly) correctly. I don't have the DNS challenge working right with Let's Encrypt, but I don't really care about that at this point. I don't really need a wildcard certificate.
I created a mariadb container. We're mostly a WordPress shop and I'd like to have one container for all the databases we work with rather than configuring a database on an environment-by-environment basis.
I created an external bridge network, named "maverick-net" and all of the stacks are connected to it.
I have a self-hosted GitHub runner listening for changes to the "dev" branch of the project. The runner pulls down the latest repo, writes GitHub secrets to a local .env file, runs composer install and then docker-compose up -d. (That's the reason behind the obscenely-long bind mount paths.)
I'm trying to make the code for these WordPress projects reusable as much as possible, so there's a lot of .env variables in the different files. At some point I'll probably move those over to docker secrets, but at this point it's a development server and not as critical.
My issue is that I haven't been able to get a WordPress site up and running, and I keep hitting a "Bad Gateway" error. When I curl the URL from inside the traefik container, I get... wait for it... "Bad Gateway."
Clearly there's something I'm missing, but I've been slamming my head against a brick wall for weeks trying different approaches to get this running and I need help. There has to be something I'm not getting about docker networks in general because my wp-cli container never has been able to connect to the database, regardless of whether I start it in the same stack or if i try to connect to it on the maverick-net network.
My traefik stack (side note, I'd really like to split these command entries into static and dynamic config files, but that's a task for another day):
version: "3.9"
secrets:
linode_token:
file: "../secrets/linode_token.secret"
services:
traefik:
container_name: traefik
image: "traefik:latest"
command:
- --log.level=DEBUG
- --log.filePath=./traefik.log
- --accessLog=true
- --accessLog.filePath=./access.log
- --accessLog.bufferingSize=100
- --accessLog.filters.statusCodes=400-499
- --api
- --api.dashboard=true
- --api.insecure=false
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --providers.docker
- --providers.docker.watch=true
- --providers.docker.exposedbydefault=false
- --certificatesresolvers.leresolver.acme.dnsChallenge=true
- --certificatesresolvers.leresolver.acme.dnsChallenge.provider=linodev4
- --certificatesresolvers.leresolver.acme.httpchallenge=true
- --certificatesresolvers.leresolver.acme.httpchallenge.entrypoint=web
- --certificatesresolvers.leresolver.acme.email=xxxxxxxxxxx#xxxxxxxxx.xxx
- --certificatesresolvers.leresolver.acme.storage=./acme.json
#- --certificatesresolvers.leresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
- --certificatesresolvers.leresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
- --experimental.hub=true
- --hub.tls.insecure=true
- --metrics.prometheus.addrouterslabels=true
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ~/certs-data/acme.json:/data/letsencrypt/acme.json
- ./static.yml:/static.yml:ro
- ./configs:/configs
- ~/certs-data/:/data/letsencrypt/
secrets:
- "linode_token"
environment:
TZ: America/Chicago
LINODE_TOKEN_FILE: "/run/secrets/linode_token"
labels:
- "traefik.enable=true"
- "traefik.docker.network=maverick-net"
- "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
- "traefik.http.routers.http-catchall.entrypoints=web"
- "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
- "traefik.http.routers.traefik.tls.certresolver=leresolver"
- "traefik.http.routers.traefik.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.traefik.entrypoints=websecure"
- "traefik.http.routers.traefik.service=api#internal"
- "traefik.http.routers.traefik.middlewares=traefik-auth"
- "traefik.http.middlewares.traefik-auth.basicauth.users=XXXX:$$apr1$$XXXXX$$XXXXXXXXXXXXXXX"
- "traefik.http.routers.api.entrypoints=websecure"
networks:
- maverick-net
hub-agent:
image: ghcr.io/traefik/hub-agent-traefik:experimental
pull_policy: always
container_name: hub-agent
restart: on-failure
command:
- run
- --hub.token=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX
- --auth-server.advertise-url=http://hub-agent
- --traefik.host=traefik
- --traefik.tls.insecure=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on:
- traefik
networks:
- maverick-net
portainer:
image: portainer/portainer-ce:latest
command: -H unix:///var/run/docker.sock
container_name: portainer
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- portainer_data:/data
labels:
# Frontend
- "traefik.enable=true"
- "traefik.http.routers.frontend.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.frontend.entrypoints=websecure"
- "traefik.http.services.frontend.loadbalancer.server.port=9000"
- "traefik.http.routers.frontend.service=frontend"
- "traefik.http.routers.frontend.tls.certresolver=leresolver"
networks:
- maverick-net
whoami:
image: "traefik/whoami"
container_name: "whoami"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls.certresolver=leresolver"
networks:
- maverick-net
volumes:
portainer_data:
networks:
maverick-net:
external: true
My mariadb stack:
version: "3"
networks:
# enable connection with Traefik
maverick-net:
external: true
services:
mariadb:
container_name: mariadb
image: mariadb:10.7
restart: always
volumes:
- "/home/xxxxxxxxxx/docker/mariadb/data:/var/lib/mysql"
expose:
- "3306"
env_file: .env
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PWD}
MYSQL_USER: ${ADMIN_DB_USER}
MYSQL_PASSWORD: ${ADMIN_DB_PWD}
networks:
- maverick-net
And finally my WordPress stack:
version: '3.8'
networks:
maverick-net:
external: true
# volumes:
# db_data:
services:
# mariadb:
# container_name: ${WORDPRESS_DB_NAME}-db
# image: mariadb:10.7
# restart: always
# volumes:
# - "db_data:/var/lib/mysql"
# env_file: .env
# environment:
# MYSQL_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
# MYSQL_USER: ${ADMIN_DB_USER}
# MYSQL_PASSWORD: ${ADMIN_DB_PWD}
wordpress:
container_name: ${WORDPRESS_DB_NAME}-wp
image: wordpress:6.0.2-fpm
volumes:
- type: bind
source: ${PROJECT_ROOT}/${WORDPRESS_DB_NAME}/${PROJECT_NAME}/${PROJECT_NAME}/wp
target: /var/www/html
restart: always
env_file: .env
environment:
WORDPRESS_DB_HOST: mariadb
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
WORDPRESS_DATABASE_USER: ${WORDPRESS_DB_USER}
WORDPRESS_DATABASE_PASSWORD: ${WORDPRESS_DB_PASSWORD}
WORDPRESS_DATABASE_NAME: ${WORDPRESS_DB_NAME}
labels:
# The labels are useful for Traefik only
- "traefik.enable=true"
- "traefik.docker.network=maverick-net"
# Get the routes from https
- "traefik.http.routers.${WORDPRESS_DB_NAME}.rule=Host(`${DEV_URL}`)"
- "traefik.http.routers.${WORDPRESS_DB_NAME}.entrypoints=websecure"
- "traefik.http.routers.${WORDPRESS_DB_NAME}.tls.certresolver=leresolver"
networks:
- maverick-net
wordpress-cli:
container_name: ${WORDPRESS_DB_NAME}-cli
image: wordpress:cli
volumes:
- type: bind
source: ${PROJECT_ROOT}/${WORDPRESS_DB_NAME}/${PROJECT_NAME}/${PROJECT_NAME}/wp
target: /var/www/html
env_file: .env
environment:
WORDPRESS_DB_HOST: mariadb
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
WORDPRESS_DATABASE_USER: ${WORDPRESS_DB_USER}
WORDPRESS_DATABASE_PASSWORD: ${WORDPRESS_DB_PASSWORD}
WORDPRESS_DATABASE_NAME: ${WORDPRESS_DB_NAME}
networks:
- maverick-net
depends_on:
- wordpress
As far as I know, you can connect to containers in the same network by using their service name.
So for example you are trying to curl to the Wordpress container from the Traefik Container.
curl 'http://wordpress/'
Should work.
In another project I use an nginx container with php-fpm.
I need to send my curl requests to the nginx container, because the php-fpm container does not handle server requests directly:
// does not work
curl 'http://php-debug/index.html'
// result
curl: (7) Failed to connect to php-debug port 80: Connection refused
// https also does not work
curl 'https://php-fpm/index.html'
// result
curl: (7) Failed to connect to php-fpm port 443: Connection refused
// This does work
curl 'http://nginx/index.html'
// result
<HTML...
For some reason https: curl requests fail, but on http I get the correct result so for local development I think it's ok.
If you are interested in a more managed solution, you can check out warden.dev. It contains a template for Wordpress too (which I use succesfully for local development). I have been using this exclusively. If you have questions how to setup WP CLI on this solution, feel free to contact me.
It comes with portainer, traefik, ssl and dns and mailhog.
Configuration is pretty straight forward, I can set up a new project within an hour and connect to the database and containers in my IDE.
https://docs.warden.dev/environments/types.html#wordpress

Connect from Angular Docker container to Asp.Net container

I created a simple program with three containers: database (MS SQL Server), backend (Asp.Net Core), and frontend (Angular 8).
To run it I use a docker-compose:
services:
sqlserver:
image: mcr.microsoft.com/mssql/server:2019-latest
#ports:
#- 1433:1433 - it's hidden
volumes:
- data-sql:/var/opt/mssql
environment:
SA_PASSWORD: "Pass"
ACCEPT_EULA: "Y"
web_api:
build:
dockerfile: WebApi/Dockerfile
#ports:
#- 5000:80 - it's hidden
depends_on:
- sqlserver
environment:
"ASPNETCORE_URLS": "http://+:5000"
"ConnectionStrings:SqlConnectionString": "Server=sqlserver,1433;Database=db;User Id=sa;Password=pass;"
web_app:
build: WebApp/
ports:
- 4200:80
depends_on:
- web_api
environment:
"ENV": "Production"
"BASE_URL": "http://web_api:5000"
I want to hide the external ports for sqlserver and web_api, because they are only used in the docker-compose services.
I could hide the sqlserver port by adding the SqlConnectionString environment to the web_api.
But this approach doesn't work with web_app. My idea was to add the "BASE_URL": "http://web_api:5000" to the web_app so it'll be able to send requests on this URL, but it doesn't work.
Have any ideas on how to do this?
This is because this kind of frontends (SPA) communicates with your backend not internally (docker network) but externally (your host network).
Accordingly, 2 steps are needed:
🔴 Backend must be accessible thru host network
🟢 Frontend should point to the public URL of backend
web_api:
build:
dockerfile: WebApi/Dockerfile
ports: # 🔴 Added
- 5000:80 # 🔴 Added
depends_on:
# ... etc
web_app:
build: WebApp/
# .... etc
environment:
"ENV": "Production"
"BASE_URL": "http://localhost:5000" # 🟢 Changed

Docker: Connection timeout when connecting to database container

I'm having an issue with my php container not connecting to my database container.
My docker-compose.yml :
version: "2"
volumes:
# this is the mysql data volume we are going to host the data inside
dev_mysql_data:
# This volume is used for elasticsearch
dev_elastic_search:
networks:
mp_pixel:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.20.0.0/16
services:
# database container for local development purposes
dev_database:
image: mysql:5.6
networks:
mp_pixel:
aliases:
- database
ports:
# port 3304 (external) is for use on your desktop sql client
# port 3306 (internal) is for use inside your application code
- 3304:3306
volumes:
# mount the mysql_data docker volume to host the local development database
- dev_mysql_data:/var/lib/mysql
# the provision file helps when trying to use the provision script to clone databases
- ./provision.cnf:/provision.cnf
environment:
MYSQL_ROOT_PASSWORD: pixel
# This is the local development version of the nginx container
dev_nginx:
image: mp-pixel-nginx:latest
build: ./nginx
ports:
- '80:80'
- '443:443'
networks:
mp_pixel:
aliases:
- nginx
depends_on:
- dev_phpfpm
volumes_from:
- dev_phpfpm
environment:
- VIRTUAL_HOST=~^(mp-pixel|mp-location|mp-feedback|mp-user|mp-phone|mp-loancalculator|mp-seo|mp-media|mp-listing|mp-development|mp-kpi|mp-newsletter|mp-auth|mp-worker|mp-search)-ph-dev.pixel.local
# This is the local development version of the phpfpm container
dev_phpfpm:
image: mp-pixel-phpfpm:latest
build:
context: ./
args:
# this build might fail, if so, run in a terminal: export SSH_KEY=$(cat ~/.ssh/id_rsa)
- SSH_KEY=$SSH_KEY
networks:
mp_pixel:
aliases:
- phpfpm
depends_on:
- dev_database
volumes:
# we override the images /www directory with the code from the live machine
- ./:/www
env_file:
# inside this file, are the shared database secrets such as username/password
- ./env/common
- ./env/dev
dev_elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.3.3
networks:
mp_pixel:
aliases:
- elasticsearch
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 1g
cap_add:
- IPC_LOCK
volumes:
- dev_elastic_search:/usr/share/elasticsearch/data
ports:
- 9200:9200
environment:
- cluster.name=dev-elasticsearch-pixel
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "xpack.security.enabled=false"
I run it with docker-compose up and the php logs show
An exception occured in driver: SQLSTATE[HY000] [2002] Connection timed out
I try to access the database container with docker exec, and I can confirm that I have the right credentials.
What could be the problem?
When your containers are up, did you already try to connect to the database with a tool like Sequel Pro? Maybe the database is just not initialized and because of this, the connection from the php container can't be established? You tried to access the db container but not the database itself.
Additionally you could add some more environment variables to the database section of your docker-compose.yml
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
- MYSQL_DATABASE=databasename
- MYSQL_USER=databaseuser
- MYSQL_PASSWORD=databasepassword
Hope that helps

Services don't communicate internally in Docker

I am trying to run an ASP.NET Core 2.0 application (REDIS + RabbitMQ + NGINX) on Docker.
When I upload these containers via docker-compose, these services work and are even accessible by Windows, since they are mapped by "HostPORT: ContainerPORT".
However, when testing the App itself, .NET informs in console that it was not possible to connect to the REDIS, for example.
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HLDGDJNAEB9E", Request id "0HLDGDJNAEB9E:00000001": An unhandled exception was thrown by the application.
StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING.
My docker-compose.yml:
version: '3'
services:
nginx:
build:
dockerfile: ./nginx/nginx.dockerfile
context: .
image: nginx
container_name: nginx
ports:
- "80:80"
networks:
- production-network
depends_on:
- "wordSearcherApp"
wordSearcherApp:
image: wordsearcherapplication
container_name: wordsearcherapp
build:
context: .
dockerfile: WordSearcher/Dockerfile
networks:
- production-network
ports:
- "61370"
volumes:
- repository:/repository
depends_on:
- redis
- rabbit
redis:
image: redis
container_name: redis
ports:
- "6379:6379"
networks:
- production-network
rabbit:
image: rabbitmq
container_name: rabbitmq
ports:
- "5672:5672"
- "15672:15672"
networks:
- production-network
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:15672"]
interval: 30s
timeout: 10s
retries: 5
networks:
production-network:
driver: bridge
volumes:
repository:
driver: local
For Connection in C#, i use this connectionString localhost:6379
How can i do this?
Thanks.
Use redis:6379 instead of localhost:6379.
Docker-Compose will use the name you've defined for a service in the docker-compose.yml file as the hostname for its container.

Make one Docker Compose service know the domain name of another

I'm using Docker Compose to create two containers. One runs an Nginx web server which serves the mydomain.com website, and the second needs to send HTTP requests to the first one (using the mydomain.com domain name).
I don't want to have to check the Nginx container's ip each time I run it and then use docker run --add-host on the second container. My goal is to run docker-compose up and that everything be ready.
I know it's not possible, but what I'm looking for is something in the line of:
# docker-compose.yml
nginx_container:
...
second_container:
extra_hosts:
# This is invalid. extra_hosts only accepts ips.
- "mydomain.com:nginx_container"
You can get a similar result using a configuration like this:
version: "3"
services:
api:
image: node:8.9.3
container_name: foo_api
domainname: api.foo.test
command: 'npm run dev'
links:
- "mongo:mongo.foo.test"
- "redis:redis.foo.test"
volumes:
- .:/app
- /app/node_modules
ports:
- "${PORT}:3000"
- "9229:9229"
depends_on:
- redis
- mongo
networks:
- backend
redis:
image: redis:3
container_name: foo_redis
domainname: redis.foo.test
ports:
- "6379:6379"
networks:
- backend
mongo:
image: mongo:3.6.2
container_name: foo_mongo
domainname: mongo.foo.test
ports:
- "${MONGO_PORT}:27017"
environment:
- MONGO_PORT=${MONGO_PORT}
networks:
- backend
networks:
backend:
driver: bridge

Resources