Docker: Connection timeout when connecting to database container - symfony

I'm having an issue with my php container not connecting to my database container.
My docker-compose.yml :
version: "2"
volumes:
# this is the mysql data volume we are going to host the data inside
dev_mysql_data:
# This volume is used for elasticsearch
dev_elastic_search:
networks:
mp_pixel:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.20.0.0/16
services:
# database container for local development purposes
dev_database:
image: mysql:5.6
networks:
mp_pixel:
aliases:
- database
ports:
# port 3304 (external) is for use on your desktop sql client
# port 3306 (internal) is for use inside your application code
- 3304:3306
volumes:
# mount the mysql_data docker volume to host the local development database
- dev_mysql_data:/var/lib/mysql
# the provision file helps when trying to use the provision script to clone databases
- ./provision.cnf:/provision.cnf
environment:
MYSQL_ROOT_PASSWORD: pixel
# This is the local development version of the nginx container
dev_nginx:
image: mp-pixel-nginx:latest
build: ./nginx
ports:
- '80:80'
- '443:443'
networks:
mp_pixel:
aliases:
- nginx
depends_on:
- dev_phpfpm
volumes_from:
- dev_phpfpm
environment:
- VIRTUAL_HOST=~^(mp-pixel|mp-location|mp-feedback|mp-user|mp-phone|mp-loancalculator|mp-seo|mp-media|mp-listing|mp-development|mp-kpi|mp-newsletter|mp-auth|mp-worker|mp-search)-ph-dev.pixel.local
# This is the local development version of the phpfpm container
dev_phpfpm:
image: mp-pixel-phpfpm:latest
build:
context: ./
args:
# this build might fail, if so, run in a terminal: export SSH_KEY=$(cat ~/.ssh/id_rsa)
- SSH_KEY=$SSH_KEY
networks:
mp_pixel:
aliases:
- phpfpm
depends_on:
- dev_database
volumes:
# we override the images /www directory with the code from the live machine
- ./:/www
env_file:
# inside this file, are the shared database secrets such as username/password
- ./env/common
- ./env/dev
dev_elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.3.3
networks:
mp_pixel:
aliases:
- elasticsearch
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 1g
cap_add:
- IPC_LOCK
volumes:
- dev_elastic_search:/usr/share/elasticsearch/data
ports:
- 9200:9200
environment:
- cluster.name=dev-elasticsearch-pixel
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "xpack.security.enabled=false"
I run it with docker-compose up and the php logs show
An exception occured in driver: SQLSTATE[HY000] [2002] Connection timed out
I try to access the database container with docker exec, and I can confirm that I have the right credentials.
What could be the problem?

When your containers are up, did you already try to connect to the database with a tool like Sequel Pro? Maybe the database is just not initialized and because of this, the connection from the php container can't be established? You tried to access the db container but not the database itself.
Additionally you could add some more environment variables to the database section of your docker-compose.yml
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
- MYSQL_DATABASE=databasename
- MYSQL_USER=databaseuser
- MYSQL_PASSWORD=databasepassword
Hope that helps

Related

Bad Gateway for WordPress containers behind Traefik reverse proxy in docker-compose

Firstly, I'd like to say that I'm not a server admin. I'm a web programmer tasked with setting up a development server and I have no idea what I'm doing. I may not be doing things according to best practice or the way you might do them. Unfortunately, with Traefik, there are 3 ways to do everything and so 2/3 of the answers that I've come across aren't compatible with my implementation and I can't figure out how to make them work. Furthermore, this isn't my only (or even primary) job duty.
Here's the setup:
Single-server docker environment on a Linode server with Ubuntu 20.04
I have one stack with Traefik, Traefik Hub, Portainer, and WhoAmI configured and working (mostly) correctly. I don't have the DNS challenge working right with Let's Encrypt, but I don't really care about that at this point. I don't really need a wildcard certificate.
I created a mariadb container. We're mostly a WordPress shop and I'd like to have one container for all the databases we work with rather than configuring a database on an environment-by-environment basis.
I created an external bridge network, named "maverick-net" and all of the stacks are connected to it.
I have a self-hosted GitHub runner listening for changes to the "dev" branch of the project. The runner pulls down the latest repo, writes GitHub secrets to a local .env file, runs composer install and then docker-compose up -d. (That's the reason behind the obscenely-long bind mount paths.)
I'm trying to make the code for these WordPress projects reusable as much as possible, so there's a lot of .env variables in the different files. At some point I'll probably move those over to docker secrets, but at this point it's a development server and not as critical.
My issue is that I haven't been able to get a WordPress site up and running, and I keep hitting a "Bad Gateway" error. When I curl the URL from inside the traefik container, I get... wait for it... "Bad Gateway."
Clearly there's something I'm missing, but I've been slamming my head against a brick wall for weeks trying different approaches to get this running and I need help. There has to be something I'm not getting about docker networks in general because my wp-cli container never has been able to connect to the database, regardless of whether I start it in the same stack or if i try to connect to it on the maverick-net network.
My traefik stack (side note, I'd really like to split these command entries into static and dynamic config files, but that's a task for another day):
version: "3.9"
secrets:
linode_token:
file: "../secrets/linode_token.secret"
services:
traefik:
container_name: traefik
image: "traefik:latest"
command:
- --log.level=DEBUG
- --log.filePath=./traefik.log
- --accessLog=true
- --accessLog.filePath=./access.log
- --accessLog.bufferingSize=100
- --accessLog.filters.statusCodes=400-499
- --api
- --api.dashboard=true
- --api.insecure=false
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --providers.docker
- --providers.docker.watch=true
- --providers.docker.exposedbydefault=false
- --certificatesresolvers.leresolver.acme.dnsChallenge=true
- --certificatesresolvers.leresolver.acme.dnsChallenge.provider=linodev4
- --certificatesresolvers.leresolver.acme.httpchallenge=true
- --certificatesresolvers.leresolver.acme.httpchallenge.entrypoint=web
- --certificatesresolvers.leresolver.acme.email=xxxxxxxxxxx#xxxxxxxxx.xxx
- --certificatesresolvers.leresolver.acme.storage=./acme.json
#- --certificatesresolvers.leresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
- --certificatesresolvers.leresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
- --experimental.hub=true
- --hub.tls.insecure=true
- --metrics.prometheus.addrouterslabels=true
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ~/certs-data/acme.json:/data/letsencrypt/acme.json
- ./static.yml:/static.yml:ro
- ./configs:/configs
- ~/certs-data/:/data/letsencrypt/
secrets:
- "linode_token"
environment:
TZ: America/Chicago
LINODE_TOKEN_FILE: "/run/secrets/linode_token"
labels:
- "traefik.enable=true"
- "traefik.docker.network=maverick-net"
- "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
- "traefik.http.routers.http-catchall.entrypoints=web"
- "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
- "traefik.http.routers.traefik.tls.certresolver=leresolver"
- "traefik.http.routers.traefik.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.traefik.entrypoints=websecure"
- "traefik.http.routers.traefik.service=api#internal"
- "traefik.http.routers.traefik.middlewares=traefik-auth"
- "traefik.http.middlewares.traefik-auth.basicauth.users=XXXX:$$apr1$$XXXXX$$XXXXXXXXXXXXXXX"
- "traefik.http.routers.api.entrypoints=websecure"
networks:
- maverick-net
hub-agent:
image: ghcr.io/traefik/hub-agent-traefik:experimental
pull_policy: always
container_name: hub-agent
restart: on-failure
command:
- run
- --hub.token=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX
- --auth-server.advertise-url=http://hub-agent
- --traefik.host=traefik
- --traefik.tls.insecure=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on:
- traefik
networks:
- maverick-net
portainer:
image: portainer/portainer-ce:latest
command: -H unix:///var/run/docker.sock
container_name: portainer
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- portainer_data:/data
labels:
# Frontend
- "traefik.enable=true"
- "traefik.http.routers.frontend.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.frontend.entrypoints=websecure"
- "traefik.http.services.frontend.loadbalancer.server.port=9000"
- "traefik.http.routers.frontend.service=frontend"
- "traefik.http.routers.frontend.tls.certresolver=leresolver"
networks:
- maverick-net
whoami:
image: "traefik/whoami"
container_name: "whoami"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`XXXXX.XXXXXXXXXX.XXX`)"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls.certresolver=leresolver"
networks:
- maverick-net
volumes:
portainer_data:
networks:
maverick-net:
external: true
My mariadb stack:
version: "3"
networks:
# enable connection with Traefik
maverick-net:
external: true
services:
mariadb:
container_name: mariadb
image: mariadb:10.7
restart: always
volumes:
- "/home/xxxxxxxxxx/docker/mariadb/data:/var/lib/mysql"
expose:
- "3306"
env_file: .env
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PWD}
MYSQL_USER: ${ADMIN_DB_USER}
MYSQL_PASSWORD: ${ADMIN_DB_PWD}
networks:
- maverick-net
And finally my WordPress stack:
version: '3.8'
networks:
maverick-net:
external: true
# volumes:
# db_data:
services:
# mariadb:
# container_name: ${WORDPRESS_DB_NAME}-db
# image: mariadb:10.7
# restart: always
# volumes:
# - "db_data:/var/lib/mysql"
# env_file: .env
# environment:
# MYSQL_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
# MYSQL_USER: ${ADMIN_DB_USER}
# MYSQL_PASSWORD: ${ADMIN_DB_PWD}
wordpress:
container_name: ${WORDPRESS_DB_NAME}-wp
image: wordpress:6.0.2-fpm
volumes:
- type: bind
source: ${PROJECT_ROOT}/${WORDPRESS_DB_NAME}/${PROJECT_NAME}/${PROJECT_NAME}/wp
target: /var/www/html
restart: always
env_file: .env
environment:
WORDPRESS_DB_HOST: mariadb
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
WORDPRESS_DATABASE_USER: ${WORDPRESS_DB_USER}
WORDPRESS_DATABASE_PASSWORD: ${WORDPRESS_DB_PASSWORD}
WORDPRESS_DATABASE_NAME: ${WORDPRESS_DB_NAME}
labels:
# The labels are useful for Traefik only
- "traefik.enable=true"
- "traefik.docker.network=maverick-net"
# Get the routes from https
- "traefik.http.routers.${WORDPRESS_DB_NAME}.rule=Host(`${DEV_URL}`)"
- "traefik.http.routers.${WORDPRESS_DB_NAME}.entrypoints=websecure"
- "traefik.http.routers.${WORDPRESS_DB_NAME}.tls.certresolver=leresolver"
networks:
- maverick-net
wordpress-cli:
container_name: ${WORDPRESS_DB_NAME}-cli
image: wordpress:cli
volumes:
- type: bind
source: ${PROJECT_ROOT}/${WORDPRESS_DB_NAME}/${PROJECT_NAME}/${PROJECT_NAME}/wp
target: /var/www/html
env_file: .env
environment:
WORDPRESS_DB_HOST: mariadb
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
WORDPRESS_DATABASE_USER: ${WORDPRESS_DB_USER}
WORDPRESS_DATABASE_PASSWORD: ${WORDPRESS_DB_PASSWORD}
WORDPRESS_DATABASE_NAME: ${WORDPRESS_DB_NAME}
networks:
- maverick-net
depends_on:
- wordpress
As far as I know, you can connect to containers in the same network by using their service name.
So for example you are trying to curl to the Wordpress container from the Traefik Container.
curl 'http://wordpress/'
Should work.
In another project I use an nginx container with php-fpm.
I need to send my curl requests to the nginx container, because the php-fpm container does not handle server requests directly:
// does not work
curl 'http://php-debug/index.html'
// result
curl: (7) Failed to connect to php-debug port 80: Connection refused
// https also does not work
curl 'https://php-fpm/index.html'
// result
curl: (7) Failed to connect to php-fpm port 443: Connection refused
// This does work
curl 'http://nginx/index.html'
// result
<HTML...
For some reason https: curl requests fail, but on http I get the correct result so for local development I think it's ok.
If you are interested in a more managed solution, you can check out warden.dev. It contains a template for Wordpress too (which I use succesfully for local development). I have been using this exclusively. If you have questions how to setup WP CLI on this solution, feel free to contact me.
It comes with portainer, traefik, ssl and dns and mailhog.
Configuration is pretty straight forward, I can set up a new project within an hour and connect to the database and containers in my IDE.
https://docs.warden.dev/environments/types.html#wordpress

Docker Compose Wordpress with pure-ftpd - FTP user has no write permissions

Kind of new to Docker, so please bear with me.
BACKGROUND:
I've setup the following containers in a Win 10 OS with WSL2:
Wordpress + MariaDB + PhpMyAdmin + Pure-ftpd
I'm not binding my project files with my OS as it slows down the whole website. Instead, I'm using pure-ftpd to update my volumes.
This setup performs great! ATM my DB is about 1GB+, Files are about 500MB and Uploads are about 22 GB. *Chef's kiss
PROBLEM:
When I create a file using FTP, this does not have "Write" permissions. So creating new scripts becomes impossible. A work around has been going to the volume and updating the file permission to "777"
pure-ftpd creates the files using user "1000", but when I try searching the user in the container, this returns nothing.
M I missing something on my .YML to allow pure-ftpd to write into the "wordpress" volume as "root"
This is my .YML
services:
#DATABASE
db:
container_name: cc_db
# We use a mariadb image which supports both amd64 & arm64 architecture
image: mariadb:10.6.9-focal
command: '--default-authentication-plugin=mysql_native_password'
volumes:
- cc_db:/var/lib/mysql/****_woo
- ./my_customized.cnf:/etc/mysql/my.cnf
ports:
- "3306:3306" # To Allow Remote Connections
restart: always
environment:
- MYSQL_ROOT_PASSWORD=******++
- MYSQL_DATABASE=******
- MYSQL_USER=wordpress
- MYSQL_PASSWORD=wordpress
expose:
- 3306
- 33060
networks:
- cc_network
#PHPMYADMIN
phpmyadmin:
image: phpmyadmin/phpmyadmin
depends_on:
- db
environment:
- UPLOAD_LIMIT=768M
- PMA_HOST:db
- PMA_PORT:3306
- PMA_ARBITRARY:1
- MYSQL_ROOT_PASSWORD=******++
restart: always
ports:
- 8080:80
networks:
- cc_network
#WORDPRESS
wordpress:
container_name: cc_wordpress
#image: wordpress:latest
# Current Website: Wordpress # 6.0.2 -- PHP 8.1.10 -- Maria DB 10.6.9 :: Post Max Size: 128 MB , PHP Limit 120 :: Max Inpt Var 4500
image: wordpress:6.0.2-php8.1
ports:
- 80:80
restart: always
networks:
- cc_network
environment:
# our local dev environment
- WORDPRESS_DEBUG:1
- WORDPRESS_DB_HOST=db:3306
- WORDPRESS_DB_USER=wordpress
- WORDPRESS_DB_PASSWORD=wordpress
- WORDPRESS_DB_NAME=*****
volumes:
- ./uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
- ./httpd/wp-config.php:/var/www/html/wp-config.php
- ./httpd/.htaccess:/var/www/html/.htaccess
- cc_wordpress:/var/www/html/wp-content:rw
ftp:
container_name: ftpd-server
image: stilliard/pure-ftpd:hardened
depends_on:
- wordpress
ports:
- 21:21
- 20:20
- 30000-30009:30000-30009
volumes:
- cc_wordpress:/home/user/:rw
- './ftp/pass:/etc/pure-ftpd/passwd'
environment:
PUBLICHOST: "10.47.61.236"
FTP_USER_NAME: "user"
FTP_USER_PASS: "*****++"
FTP_USER_HOME: "/home/user"
ADDED_FLAGS: "--tls=2"
TLS_CN: "**** FTP"
TLS_ORG: "*****"
TLS_C: "US"
MAX_CONNECTIONS: "20"
restart: always
networks:
- cc_network
networks:
cc_network:
volumes:
cc_wordpress:
cc_db:
According to the pure-ftpd documentation you could indicate the UID and GID of the FTP user using the appropriate environment variables:
If you wish to set the UID & GID of the FTP user, use the FTP_USER_UID & FTP_USER_GID environment variables.
The documentation provides as well an example of using pure-ftpd explicitly with Wordpress. It mentions:
In the Wordpress container, the owner of the files has the UID 33 & GID 33, thus we set the UID & GID of the FTP user accordingly, providing the following code snipplet:
version: "3.2"
services:
web:
image: wordpress:4.8-apache
# other configs for wordpress
volumes:
- ./data/wordpress:/var/www/html
ftp:
# optionally replace username/repo:tag with your name and image details
image: stilliard/pure-ftpd:latest
deploy:
replicas: 1
restart_policy:
condition: on-failure
environment:
PUBLICHOST: xxx.xxx.xxx.xxx
FTP_USER_NAME: "bob"
FTP_USER_PASS: "foobarqux"
FTP_USER_HOME: "/var/www/html"
FTP_USER_UID: 33
FTP_USER_GID: 33
volumes:
- ./data/wordpress:/var/www/html
- ./data/ftp:/etc/ssl/private
ports:
- target: 21
published: 21
protocol: tcp
mode: host
# Bind each passive ports to the host
- target: 30000
published: 30000
protocol: tcp
mode: host
# ...
Please, try modifying your docker-compose file accordingly, I suppose something similar to this:
services:
#DATABASE
db:
container_name: cc_db
# We use a mariadb image which supports both amd64 & arm64 architecture
image: mariadb:10.6.9-focal
command: '--default-authentication-plugin=mysql_native_password'
volumes:
- cc_db:/var/lib/mysql/****_woo
- ./my_customized.cnf:/etc/mysql/my.cnf
ports:
- "3306:3306" # To Allow Remote Connections
restart: always
environment:
- MYSQL_ROOT_PASSWORD=******++
- MYSQL_DATABASE=******
- MYSQL_USER=wordpress
- MYSQL_PASSWORD=wordpress
expose:
- 3306
- 33060
networks:
- cc_network
#PHPMYADMIN
phpmyadmin:
image: phpmyadmin/phpmyadmin
depends_on:
- db
environment:
- UPLOAD_LIMIT=768M
- PMA_HOST:db
- PMA_PORT:3306
- PMA_ARBITRARY:1
- MYSQL_ROOT_PASSWORD=******++
restart: always
ports:
- 8080:80
networks:
- cc_network
#WORDPRESS
wordpress:
container_name: cc_wordpress
#image: wordpress:latest
# Current Website: Wordpress # 6.0.2 -- PHP 8.1.10 -- Maria DB 10.6.9 :: Post Max Size: 128 MB , PHP Limit 120 :: Max Inpt Var 4500
image: wordpress:6.0.2-php8.1
ports:
- 80:80
restart: always
networks:
- cc_network
environment:
# our local dev environment
- WORDPRESS_DEBUG:1
- WORDPRESS_DB_HOST=db:3306
- WORDPRESS_DB_USER=wordpress
- WORDPRESS_DB_PASSWORD=wordpress
- WORDPRESS_DB_NAME=*****
volumes:
- ./uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
- ./httpd/wp-config.php:/var/www/html/wp-config.php
- ./httpd/.htaccess:/var/www/html/.htaccess
- cc_wordpress:/var/www/html/wp-content:rw
ftp:
container_name: ftpd-server
image: stilliard/pure-ftpd:hardened
depends_on:
- wordpress
ports:
- 21:21
- 20:20
- 30000-30009:30000-30009
volumes:
- cc_wordpress:/home/user/:rw
- './ftp/pass:/etc/pure-ftpd/passwd'
environment:
PUBLICHOST: "10.47.61.236"
FTP_USER_NAME: "user"
FTP_USER_PASS: "*****++"
FTP_USER_HOME: "/home/user"
FTP_USER_UID: 33
FTP_USER_GID: 33
ADDED_FLAGS: "--tls=2"
TLS_CN: "**** FTP"
TLS_ORG: "*****"
TLS_C: "US"
MAX_CONNECTIONS: "20"
restart: always
networks:
- cc_network
networks:
cc_network:
volumes:
cc_wordpress:
cc_db:
You are using the wordpress:6.0.2-php8.1 image which in turn is based on php:8.1-apache. As far I understand from the php:8.1-apache Dockerfile you would need to adjust the FTP_USER_UID and FTP_USER_GID variables to match the ones used to run Apache, I assume, the user www-data, created by default in Debian systems with UID and GID 33.
Excellent answer by #jccampanero. Would like to add that you can docker exec something like pure-pw useradd yourusername -f /etc/pure-ftpd/passwd/pureftpd.passwd -m -u ftpuser -d /home/ftpusers/youruser. Also according to documentation,
If you have any trouble with volume permissions due to the uid or gid
of the created user you can change the -u flag for the uid you would
like to use and/or specify -g with the group id as well. For more
information see issue

Host-local networks in Docker Compose

I have the following docker-compose.yml, but need to model a public/private network split where the Redis instance must only be accessible to localhost.
version: "2.2" # for compatibility, I can upgrade if needed
services:
nginx:
image: nginx
ports:
- 8080:80
redis:
image: redis
ports:
- 6379:6379
This seems straightforward if I needed to restrict it to being accessible only within the docker network. Consider:
version: "2.2"
services:
nginx:
image: nginx
ports:
- 8080:80
networks:
- frontend
- backend
redis:
image: redis
ports:
- 6379:6379
networks:
- backend
networks:
frontend: {}
backend:
internal: true
However our local web developers need to be able to access that Redis instance from their host machine (outside of the docker network) when they build, run, and debug locally.
Just bind the service port of redis to localhost(127.0.0.1).
Try follows...
...
redis:
image: redis
ports:
- 127.0.0.1:6379:6379
...
Run Redis Web UI called redis-commander.
Use env vars to point to the running redis.
Expose this new container & access it instead of exposing Redis container.
services:
redis:
# Do comment ports! no need to expose it
# ports:
# - 6379:6379
// ....
redis-commander:
image: rediscommander/redis-commander:latest
restart: unless-stopped
environment:
REDIS_HOST: redis:6379 # <-- 🔴 Here you point to your redis
# REDIS_PASSWORD # <- in case redis is protected with password
ports:
- "8081:8081"
Let your developers go to http://localhost:8081 and enjoy.
Find more details about that image

Transfer symfony logfiles with filebeat to graylog in local docker-environment

Description
I am trying to build an equal configuration in my local docker-environment like on our production system. After spending some time investigating and rebuilding the docker container setup, still can't get it to work and Graylog is not receiving any data.
Overview and interim results
web, php and db container are in use for the symfony based application
symfony runs properly on localhost in php-container and generates logfiles
symfony-logfiles are located here: /var/www/html/var/logs/*.log
symfony-logfiles format is json / gelf
all other containers are also up and running when starting the complete composition
filebeat configuration is based on first link below
filebeat.yml seems to retrieve any logfile found in any container
filebeat configured to transfer data directly to elasticsearch
elasticsearch persists data in mongodb
all graylog related data in persisted in named volumes in docker
additionally I am working with docker-sync on a Mac
The docker-compose.yml is based on the following resources:
https://github.com/jochenchrist/docker-logging-elasticsearch
http://docs.graylog.org/en/2.4/pages/installation/docker.html?highlight=docker
https://www.elastic.co/guide/en/beats/filebeat/6.3/running-on-docker.html
https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-reference-yml.html
config.yml
# Monolog Configuration
monolog:
channels: [graylog]
handlers:
graylog:
type: stream
formatter: line_formatter
path: "%kernel.logs_dir%/graylog.log"
channels: [graylog]
docker-compose.yml
version: "3"
services:
web:
image: nginx
ports:
- "80:80"
- "443:443"
links:
- php
volumes:
- ./docker-config/nginx.conf:/etc/nginx/conf.d/default.conf
- project-app-sync:/var/www/html
- ./docker-config/localhost.crt:/etc/nginx/ssl/localhost.crt
- ./docker-config/localhost.key:/etc/nginx/ssl/localhost.key
php:
build:
context: .
dockerfile: ./docker-config/Dockerfile-php
links:
- graylog
volumes:
- project-app-sync:/var/www/html
- ./docker-config/php.ini:/usr/local/etc/php/php.ini
- ./docker-config/www.conf:/usr/local/etc/php-fpm.d/www.conf
db:
image: mysql
ports:
- "3306:3306"
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
- MYSQL_DATABASE=project
- MYSQL_USER=project
- MYSQL_PASSWORD=password
volumes:
- ./docker-config/mysql.cnf:/etc/mysql/conf.d/mysql.cnf
- project-mysql-sync:/var/lib/mysql
# Graylog / Filebeat
filebeat:
build: ./docker-config/filebeat
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
networks:
- graylog-network
depends_on:
- graylog-elasticsearch
graylog:
image: graylog/graylog:2.4
volumes:
- graylog-journal:/usr/share/graylog/data/journal
networks:
- graylog-network
environment:
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
links:
- graylog-mongo:mongo
- graylog-elasticsearch:elasticsearch
depends_on:
- graylog-mongo
- graylog-elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
graylog-mongo:
image: mongo:3
volumes:
- graylog-mongo-data:/data/db
networks:
- graylog-network
graylog-elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.10
ports:
- "9200:9200"
volumes:
- graylog-elasticsearch-data:/usr/share/elasticsearch/data
networks:
- graylog-network
environment:
- cluster.name=graylog
- "discovery.zen.minimum_master_nodes=1"
- "discovery.type=single-node"
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
# Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
- xpack.security.enabled=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
project-app-sync:
external: true
project-mysql-sync: ~
graylog-mongo-data:
driver: local
graylog-elasticsearch-data:
driver: local
graylog-journal:
driver: local
networks:
graylog-network: ~
Dockerfile of filebeat container
FROM docker.elastic.co/beats/filebeat:6.3.1
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
# must run as root to access /var/lib/docker and /var/run/docker.sock
USER root
RUN chown root /usr/share/filebeat/filebeat.yml
# dont run with -e, to disable output to stderr
CMD [""]
filebeat.yml
filebeat.prospectors:
- type: docker
paths:
- '/var/lib/docker/containers/*/*.log'
# path to symfony based logs
- '/var/www/html/var/logs/*.log'
containers.ids: '*'
processors:
- decode_json_fields:
fields: ["host","application","short_message"]
target: ""
overwrite_keys: true
- add_docker_metadata: ~
output.elasticsearch:
# transfer data to elasticsearch container?
hosts: ["localhost:9200"]
logging.to_files: true
logging.to_syslog: false
Graylog backend
After setting up this docker composition I started the Graylog web-view and set up a collector and input as described here:
http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#step-by-step-guide
Maybe I have totally misunderstood how this could work. I am not totally sure if Beats from Elastic is the same as the filebeats container and if the sidecar collector is something extra I forgot to add. Maybe I misconfigured the collector and input in graylog?!
I would be thankful to any help or working example according to my problem ...
Graylog seems to be running on http://127.0.0.1:9000/api which is in the container. You might want to run it as http://graylog:9000/api or as http://0.0.0.0:9000/api
Accessing the other images from within any of the other images will have be done with the same name as the service name, as defined in the docker-compose.yml files. The url to the graylog-elasticsearch would be something like: http://graylog-elasticsearch/.... if you would post to localhost it would stay inside its own image.
Hope this will help you along in finding the solution.

Make one Docker Compose service know the domain name of another

I'm using Docker Compose to create two containers. One runs an Nginx web server which serves the mydomain.com website, and the second needs to send HTTP requests to the first one (using the mydomain.com domain name).
I don't want to have to check the Nginx container's ip each time I run it and then use docker run --add-host on the second container. My goal is to run docker-compose up and that everything be ready.
I know it's not possible, but what I'm looking for is something in the line of:
# docker-compose.yml
nginx_container:
...
second_container:
extra_hosts:
# This is invalid. extra_hosts only accepts ips.
- "mydomain.com:nginx_container"
You can get a similar result using a configuration like this:
version: "3"
services:
api:
image: node:8.9.3
container_name: foo_api
domainname: api.foo.test
command: 'npm run dev'
links:
- "mongo:mongo.foo.test"
- "redis:redis.foo.test"
volumes:
- .:/app
- /app/node_modules
ports:
- "${PORT}:3000"
- "9229:9229"
depends_on:
- redis
- mongo
networks:
- backend
redis:
image: redis:3
container_name: foo_redis
domainname: redis.foo.test
ports:
- "6379:6379"
networks:
- backend
mongo:
image: mongo:3.6.2
container_name: foo_mongo
domainname: mongo.foo.test
ports:
- "${MONGO_PORT}:27017"
environment:
- MONGO_PORT=${MONGO_PORT}
networks:
- backend
networks:
backend:
driver: bridge

Resources