I have 2 folders separated, one for backend and one for frontend services:
backend/docker-compose.yml
frontend/docker-compose.yml
The backend has a headless wordpress installation on nginx, with the scope to serve the frontend as an api service. The frontend runs on next.js. Here are the 2 different docker-compose.yml:
backend/docker-compose.yml
version: '3.9'
services:
nginx:
image: nginx:latest
container_name: my-app-nginx
ports:
- '80:80'
- '443:443'
- '8080:8080'
...
networks:
- internal-network
mysql:
...
networks:
- internal-network
wordpress:
...
networks:
- internal-network
networks:
internal-network:
external: true
frontend/docker-compose.yml
version: '3.9'
services:
nextjs:
build:
...
container_name: my-app-nextjs
restart: always
ports:
- 3000:3000
networks:
- internal-network
networks:
internal-network:
driver: bridge
name: internal-network
In the frontend I use the fetch api in nextjs as following:
fetch('http://my-app-nginx/wp-json/v1/enpoint', ...)
I tried also with ports 80 and 8080, without success.
The sequence of commands I run are:
docker network create internal-network
in backend/ folder, docker-compose up -d (all backend containers run fine, I can fetch data with Postman from WordPress api)
in frontend/ folder, docker-compose up -d fails with the error Error: getaddrinfo EAI_AGAIN my-app-nginx
I am not a very expert user of docker so I might miss something here, but I understand that there might be internal network issues over the containers. I read many answers regarding this topic but I couldn't figure it out.
Any recommendations?
Just to add a proper answer:
Generally you should NOT really want to be executing multiple docker-compose up -d commands
If you want to combine two separate docker-compose configs and run as one (slightly more preferable), you can use the extends keyword as described in the docs
However, I would suggest that you treat it as a single docker-compose project which can itself have multiple nested git repositories:
Example SO answer - Git repository setup for a Docker application consisting of multiple repositories
You can keep your code in a mono-repo or multiple repos, up to you
Real working example to backup using your applications that validates this approach:
headless-wordpress-nextjs-starter-kit and it's docker-compose.yml
I have found this thread here
Communication between multiple docker-compose projects
By looking at the most upvoted answers, I wonder if it is related to network prefix?
It seems like the internal-network would be prefixed with frontend_? On the other hand you can also try to locate the network by name in backend/docker-compose.yml:
networks:
internal-network:
external:
name: internal-network
The issue is external networks need the network name specified (because docker compose prefixes resources by default). Your backend docker compose network section should look like this:
networks:
internal-network:
name: internal-network
external: true
You are creating the network in your frontend docker compose so you should omit the docker network create ... command (just need to init frontend first). Or instead treat them both as external and keep the command. In which use the named external network as shown above in your frontend docker compose as well.
Issue
I've a set of working WordPress Docker Compose containers that includes blog image and db image.
version: '3'
services:
wordpress-app-db:
build:
context: ./mysql
image: wordpress-app-db:5.7-mysql
restart: always
php7-wordpress-app:
depends_on:
- wordpress-app-db
build:
context: ./wordpress
image: wordpress-app:php7
links:
- wordpress-app-db
ports:
- "8080:80"
restart: always
volumes:
data:
Now, here the above yaml will work with no issues at all, but when I want to change the port 8080 to some other port it simply wont work.
ports:
- "<my-custom-port>:80"
All around the url takes me to http://localhost:8080/.
I'm confused of it's behaviour, I'm not able to understand why it is redirecting to 8080 if it has been mapped to other port <my-custom-port>.
For info, I've exposed port 80 in Dockerfile.
Reason
I want to do so, as I've to run this set in kubernetes cluster with nodePort and I can't assign it the port 8080 in nodePort.
Verify that the siteurl and home wp_options in your database are set to the correct hostname and port. I recently migrated a WordPress site from a LAMP compute instance to Kubernetes using the official WordPress images, and I experienced the exact same issue.
WordPress will perform a redirect if those values do not match your domain name.
In my case, the two fields mutated somehow during the migration, causing every request to redirect to :8080. They were changed to localhost:8080.
Have you cross checked the target port in your service file?
There would be 3 entries in service file.
containerPort which maps to internal redirect something like custom port which you use in your docker-compose
targetPort which maps to actual application port it is listening to
nodePort which is one which you can access using <http/s>://nodeIP:nodePort
I am kind of in over my head with my current small project.
(although it should not be that hard)
I am trying to run multiple webpages using docker on my Pi (for testing purposes) which should all be reachable using the PI's IP.
I currently run a minimL LIGHTTPD: (based on the resin/rpi-raspbian image)
docker run -d -v <testconfig>:/etc/lighttpd -p <pi-ip>:8080:80 <image name>
(this server is reachable using the browser on pi and on other computers in the network)
For nginx I run another container with with a simple config
(starting with http://nginx.org/en/docs/beginners_guide.html),
containing a webpage and images to test the container config.
this container is reachable using <pi-ip>:80
then I tried to add a proxy to the locations:
(I played around so now there are 3 locations for the same redirect)
location /prox1/{
proxy_pass http://<pi-ip>:8080
}
location /prox2/{
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://<pi-ip>:8080
}
location /prox3/{
fastcgi_pass <pi-ip>:8080;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string
}
Version 1&2 give a 404 (I tried adding a rewrite, but then I ´nginx redirected on itself due to the /prox1/ being cut).
Version 3 yields a timeout.
Now I am not sure if I still have to dig on the nginx side, or I have to add a connection on the docker side between the containers.
PS: the Pi is running ArchForArm (using Xfce as desktop) because I couldn't find docker-compose in the raspberian repository.
-- EDIT ---:
I currently start everything manually. (so no compose file)
the LIGHTTPD is started with:
docker run -d --name mylighttpd -v <testconfig>:/etc/lighttpd -p <pi-ip>:8080:80 <image name>
if I understood it correctly it is now listening on the local network (in the range of <pi-ip>) port 8080, which represents the test web-servers port 80. (I have added ..name so it is easier to stop it.)
the nginx is started like:
docker run --name mynginx --rm -p <pi-ip>:80:80 -v <config>:/data <image name>
The 8080 was added in the expose in the Docker file.
I current think I misunderstood the connection for two clients on the same machine, and should add a Virtual network, I am currently trying to find some docks there.
PS: I am not using the already existing nginx-zeroconf from the repo because it tells me it cant read the installed docker version. (and the only example for using that with composer also needs another container which seems unavailable for my architecture.)
-- edit2 --:
For the simple proxy_pass the problem could be the URL.
I added a deeper folder "prox1" in the "www" folder, containing an index file, and that one is schown when i ask for the page.
It seems like <pi-ip>:80/prox1/
is redirected to <pi-ip>:8080/prox1/
but if I try rewrite it (inside "location /prox1/") it seems to first delete the prox1, and then decides it now is part of the original location.
<pi-ip>:80/
PS: I am aware that it might be a better design to place the system inside another connection than "bridge" and only expose the proxy, but i am trying to learn this stuff in small steps.
-- edit3 --:
Trying compose now, but it seems I have encounters another part I don't understand (why I wanted to get it work without compose first).
I try to follow http://docs.master.dockerproject.org/compose/compose-file/#ipv4-address-ipv6-address
networks:
backbone:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.238.0/16
gateway: 172.16.238.1
services:
nginx:
image: <nginx-image>
ports: 80:80
volumes:
- <config>:/data
depends_on:
- lighttpd
networks:
backbone:
ipv4_address: 172.16.238.2
lighttpd:
image: <lighttpd-image>
ports: 8080:80
volumes:
- <testconfig>:/etc/lighttpd
networks:
backbone:
ipv4_address: 172.16.238.3
Now I have to find out why i get "User specific IP address is supported only when connecting to networks with user configured subnets", I assume the main networks block creates a network called "backbone".
-- edit4 --:
It seems ip blocks have to be written different to all the docks I have seen, the correct form is:
...
networks:
backbone:
ipv4_address: 172.16.0.2/16
...
now I have to figure out how to drop the part of the URL, and I am good to go.
The core problem seems to have been missing nginx parameter proxy_redirect, i found rambling trough the docks, the current nginx.conf is:
(/data/www contains a index.html with a relative link to some images in /data/images)
worker_processes auto;
events {
worker_connections 1024;
}
http {
server {
listen 80;
location / {
root /data/www;
}
location /images/ {
root /data;
}
location /prox0/{
proxy_pass http://lighttpd:80;
proxy_redirect default;
proxy_buffering off;
}
}
}
manual starting on local Ip seems to work, but docker-compose is easyer:
(if compose is not used replace lighttpd:80 with the ip & port used for starting the server.)
networks:
backbone:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.238.0/16
gateway: 172.16.238.1
services:
nginx:
image: <nginx-image>
ports: 80:80
volumes:
- <config>:/data
depends_on:
- lighttpd
networks:
backbone:
ipv4_address: 172.16.0.2
lighttpd:
image: <lighttpd-image>
ports: 8080:80
volumes:
- <testconfig>:/etc/lighttpd
networks:
backbone:
ipv4_address: 172.16.0.3
I have a webserver that requires websocket connection in production. I deploy it using docker-compose with nginx as proxy.
So my compose file look like this:
version: '2'
services:
app:
restart: always
nginx:
restart: always
ports:
- "80:80"
Now if I scale "app" service to multiple instances, docker-compose will perform round robin on each call to the internal dns "app".
Is there a way to tell docker-compose load balancer to apply sticky sessions?
Another solution - is there a way to solve it using nginx?
Possible solution that I don't like:
multiple definitions of app
version: '2'
services:
app1:
restart: always
app2:
restart: always
nginx:
restart: always
ports:
- "80:80"
(And then on nginx config file I can define sticky sessions between app1 and app2).
Best result I got from searching:
https://github.com/docker/dockercloud-haproxy
But this requires me to add another service (maybe replace nginx?) and the docs is pretty poor about sticky sessions there.
I wish docker would just allow configuring it with simple line in the compose file.
Thanks!
Take a look at jwilder/nginx-proxy. This image provides an nginx reverse proxy that listens for containers that define the VIRTUAL_HOST variable and automatically updates its configuration on container creation and removal. tpcwang's fork allows you to use the IP_HASH directive on a container level to enable sticky sessions.
Consider the following Compose file:
nginx:
image: tpcwang/nginx-proxy
ports:
- "80:80"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
app:
image: tutum/hello-world
environment:
- VIRTUAL_HOST=<your_ip_or_domain_name>
- USE_IP_HASH=1
Let's get it up and running and then scale app to three instances:
docker-compose up -d
docker-compose scale app=3
If you check the nginx configuration file you'll see something like this:
docker-compose exec nginx cat /etc/nginx/conf.d/default.conf
...
upstream 172.16.102.132 {
ip_hash;
# desktop_app_3
server 172.17.0.7:80;
# desktop_app_2
server 172.17.0.6:80;
# desktop_app_1
server 172.17.0.4:80;
}
server {
server_name 172.16.102.132;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
location / {
proxy_pass http://172.16.102.132;
}
}
The nginx container has automatically detected the three instances and has updated its configuration to route requests to all of them using sticky sessions.
If we try to access the app we can see that it always reports the same hostname on each refresh. If we remove the USE_IP_HASH environment variable we'll see that the hostname actually changes, this is, the nginx proxy is using round robin to balance our requests.
I have recently started migrating to Docker 1.9 and Docker-Compose 1.5's networking features to replace using links.
So far with links there were no problems with nginx connecting to my php5-fpm fastcgi server located in a different server in one group via docker-compose. Newly though when I run docker-compose --x-networking up my php-fpm, mongo and nginx containers boot up, however nginx quits straight away with [emerg] 1#1: host not found in upstream "waapi_php_1" in /etc/nginx/conf.d/default.conf:16
However, if I run the docker-compose command again while the php and mongo containers are running (nginx exited), nginx starts and works fine from then on.
This is my docker-compose.yml file:
nginx:
image: nginx
ports:
- "42080:80"
volumes:
- ./config/docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
php:
build: config/docker/php
ports:
- "42022:22"
volumes:
- .:/var/www/html
env_file: config/docker/php/.env.development
mongo:
image: mongo
ports:
- "42017:27017"
volumes:
- /var/mongodata/wa-api:/data/db
command: --smallfiles
This is my default.conf for nginx:
server {
listen 80;
root /var/www/test;
error_log /dev/stdout debug;
access_log /dev/stdout;
location / {
# try to serve file directly, fallback to app.php
try_files $uri /index.php$is_args$args;
}
location ~ ^/.+\.php(/|$) {
# Referencing the php service host (Docker)
fastcgi_pass waapi_php_1:9000;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;
# We must reference the document_root of the external server ourselves here.
fastcgi_param SCRIPT_FILENAME /var/www/html/public$fastcgi_script_name;
fastcgi_param HTTPS off;
}
}
How can I get nginx to work with only a single docker-compose call?
This can be solved with the mentioned depends_on directive since it's implemented now (2016):
version: '2'
services:
nginx:
image: nginx
ports:
- "42080:80"
volumes:
- ./config/docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
depends_on:
- php
php:
build: config/docker/php
ports:
- "42022:22"
volumes:
- .:/var/www/html
env_file: config/docker/php/.env.development
depends_on:
- mongo
mongo:
image: mongo
ports:
- "42017:27017"
volumes:
- /var/mongodata/wa-api:/data/db
command: --smallfiles
Successfully tested with:
$ docker-compose version
docker-compose version 1.8.0, build f3628c7
Find more details in the documentation.
There is also a very interesting article dedicated to this topic: Controlling startup order in Compose
There is a possibility to use "volumes_from" as a workaround until depends_on feature (discussed below) is introduced. All you have to do is change your docker-compose file as below:
nginx:
image: nginx
ports:
- "42080:80"
volumes:
- ./config/docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
volumes_from:
- php
php:
build: config/docker/php
ports:
- "42022:22"
volumes:
- .:/var/www/html
env_file: config/docker/php/.env.development
mongo:
image: mongo
ports:
- "42017:27017"
volumes:
- /var/mongodata/wa-api:/data/db
command: --smallfiles
One big caveat in the above approach is that the volumes of php are exposed to nginx, which is not desired. But at the moment this is one docker specific workaround that could be used.
depends_on feature
This probably would be a futuristic answer. Because the functionality is not yet implemented in Docker (as of 1.9)
There is a proposal to introduce "depends_on" in the new networking feature introduced by Docker. But there is a long running debate about the same # https://github.com/docker/compose/issues/374 Hence, once it is implemented, the feature depends_on could be used to order the container start-up, but at the moment, you would have to resort to one of the following:
make nginx retry until the php server is up - I would prefer this one
use volums_from workaround as described above - I would avoid using this, because of the volume leakage into unnecessary containers.
If you are so lost for read the last comment. I have reached another solution.
The main problem is the way that you named the services names.
In this case, if in your docker-compose.yml, the service for php are called "api" or something like that, you must ensure that in the file nginx.conf the line that begins with fastcgi_pass have the same name as the php service. i.e fastcgi_pass api:9000;
Lets say the php service name is php_service then the code will be:
In the file docker-compose.yml
php_service:
build:
dockerfile: ./docker/php/Dockerfile
In the file nginx.conf
location ~ \.php$ {
fastcgi_pass php_service:9000;
fastcgi_param SCRIPT_FILENAME$document_root$fastcgi_script_name;
include fastcgi_params;
}
You can set the max_fails and fail_timeout directives of nginx to indicate that the nginx should retry the x number of connection requests to the container before failing on the upstream server unavailability.
You can tune these two numbers as per your infrastructure and speed at which the whole setup is coming up. You can read more details about the health checks section of the below URL:
http://nginx.org/en/docs/http/load_balancing.html
Following is the excerpt from http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server
max_fails=number
sets the number of unsuccessful attempts to communicate with the
server that should happen in the duration set by the fail_timeout
parameter to consider the server unavailable for a duration also set
by the fail_timeout parameter. By default, the number of unsuccessful
attempts is set to 1. The zero value disables the accounting of
attempts. What is considered an unsuccessful attempt is defined by the
proxy_next_upstream, fastcgi_next_upstream, uwsgi_next_upstream,
scgi_next_upstream, and memcached_next_upstream directives.
fail_timeout=time
sets the time during which the specified number of unsuccessful
attempts to communicate with the server should happen to consider the
server unavailable; and the period of time the server will be
considered unavailable. By default, the parameter is set to 10
seconds.
To be precise your modified nginx config file should be as follows (this script is assuming that all the containers are up by 25 seconds at least, if not, please change the fail_timeout or max_fails in below upstream section):
Note: I didn't test the script myself, so you could give it a try!
upstream phpupstream {
server waapi_php_1:9000 fail_timeout=5s max_fails=5;
}
server {
listen 80;
root /var/www/test;
error_log /dev/stdout debug;
access_log /dev/stdout;
location / {
# try to serve file directly, fallback to app.php
try_files $uri /index.php$is_args$args;
}
location ~ ^/.+\.php(/|$) {
# Referencing the php service host (Docker)
fastcgi_pass phpupstream;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;
# We must reference the document_root of the external server ourselves here.
fastcgi_param SCRIPT_FILENAME /var/www/html/public$fastcgi_script_name;
fastcgi_param HTTPS off;
}
}
Also, as per the following Note from docker (https://github.com/docker/docker.github.io/blob/master/compose/networking.md#update-containers), it is evident that the retry logic for checking the health of the other containers is not docker's responsibility and rather the containers should do the health check themselves.
Updating containers
If you make a configuration change to a service and run docker-compose
up to update it, the old container will be removed and the new one
will join the network under a different IP address but the same name.
Running containers will be able to look up that name and connect to
the new address, but the old address will stop working.
If any containers have connections open to the old container, they
will be closed. It is a container's responsibility to detect this
condition, look up the name again and reconnect.
My problem was that I forgot to specify network alias in
docker-compose.yml in php-fpm
networks:
- u-online
It is works well!
version: "3"
services:
php-fpm:
image: php:7.2-fpm
container_name: php-fpm
volumes:
- ./src:/var/www/basic/public_html
ports:
- 9000:9000
networks:
- u-online
nginx:
image: nginx:1.19.2
container_name: nginx
depends_on:
- php-fpm
ports:
- "80:8080"
- "443:443"
volumes:
- ./docker/data/etc/nginx/conf.d/default.conf:/etc/nginx/conf.d/default.conf
- ./docker/data/etc/nginx/nginx.conf:/etc/nginx/nginx.conf
- ./src:/var/www/basic/public_html
networks:
- u-online
#Docker Networks
networks:
u-online:
driver: bridge
I believe Nginx dont take in account Docker resolver (127.0.0.11), so please, can you try adding:
resolver 127.0.0.11
in your nginx configuration file?
I had the same problem because there was two networks defined in my docker-compose.yml: one backend and one frontend.
When I changed that to run containers on the same default network everything started working fine.
I found solution for services, that may be disabled for local development. Just use variables, that prevents emergency shutdown and works after service is available.
server {
location ^~ /api/ {
# other config entries omitted for breavity
set $upstream http://api.awesome.com:9000;
# nginx will now start if host is not reachable
fastcgi_pass $upstream;
fastcgi_index index.php;
}
}
source: https://sandro-keil.de/blog/let-nginx-start-if-upstream-host-is-unavailable-or-down/
Had the same problem and solved it. Please add the following line to docker-compose.yml nginx section:
links:
- php:waapi_php_1
Host in nginx config fastcgi_pass section should be linked inside docker-compose.yml nginx configuration.
At the first glance, I missed, that my "web" service didn't actually start, so that's why nginx couldn't find any host
web_1 | python3: can't open file '/var/www/app/app/app.py': [Errno 2] No such file or directory
web_1 exited with code 2
nginx_1 | [emerg] 1#1: host not found in upstream "web:4044" in /etc/nginx/conf.d/nginx.conf:2
Two things worth to mention:
Using same network bridge
Using links to add hosts resol
My example:
version: '3'
services:
mysql:
image: mysql:5.7
restart: always
container_name: mysql
volumes:
- ./mysql-data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: tima#123
network_mode: bridge
ghost:
image: ghost:2
restart: always
container_name: ghost
depends_on:
- mysql
links:
- mysql
environment:
database__client: mysql
database__connection__host: mysql
database__connection__user: root
database__connection__password: xxxxxxxxx
database__connection__database: ghost
url: https://www.itsfun.tk
volumes:
- ./ghost-data:/var/lib/ghost/content
network_mode: bridge
nginx:
image: nginx
restart: always
container_name: nginx
depends_on:
- ghost
links:
- ghost
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
- ./nginx/conf.d:/etc/nginx/conf.d
- ./nginx/letsencrypt:/etc/letsencrypt
network_mode: bridge
If you don't specify a special network bridge, all of them will use the same default one.
Add the links section to your nginx container configuration.
You have to make visible the php container to the nginx container.
nginx:
image: nginx
ports:
- "42080:80"
volumes:
- ./config/docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
links:
- php:waapi_php_1
With links there is an order of container startup being enforced. Without links the containers can start in any order (or really all at once).
I think the old setup could have hit the same issue, if the waapi_php_1 container was slow to startup.
I think to get it working, you could create an nginx entrypoint script that polls and waits for the php container to be started and ready.
I'm not sure if nginx has any way to retry the connection to the upstream automatically, but if it does, that would be a better option.
You have to use something like docker-gen to dynamically update nginx configuration when your backend is up.
See:
https://hub.docker.com/r/jwilder/docker-gen/
https://github.com/jwilder/nginx-proxy
I believe Nginx+ (premium version) contains a resolve parameter too (http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream)
Perhaps the best choice to avoid linking containers issues are the docker networking features
But to make this work, docker creates entries in the /etc/hosts for each container from assigned names to each container.
with docker-compose --x-networking -up is something like
[docker_compose_folder]-[service]-[incremental_number]
To not depend on unexpected changes in these names you should use the parameter
container_name
in your docker-compose.yml as follows:
php:
container_name: waapi_php_1
build: config/docker/php
ports:
- "42022:22"
volumes:
- .:/var/www/html
env_file: config/docker/php/.env.development
Making sure that it is the same name assigned in your configuration file for this service. I'm pretty sure there are better ways to do this, but it is a good approach to start.
My Workaround (after much trial and error):
In order to get around this issue, I had to get the full name of the 'upstream' Docker container, found by running docker network inspect my-special-docker-network and getting the full name property of the upstream container as such:
"Containers": {
"39ad8199184f34585b556d7480dd47de965bc7b38ac03fc0746992f39afac338": {
"Name": "my_upstream_container_name_1_2478f2b3aca0",
Then used this in the NGINX my-network.local.conf file in the location block of the proxy_pass property: (Note the addition of the GUID to the container name):
location / {
proxy_pass http://my_upsteam_container_name_1_2478f2b3aca0:3000;
As opposed to the previously working, but now broken:
location / {
proxy_pass http://my_upstream_container_name_1:3000
Most likely cause is a recent change to Docker Compose, in their default naming scheme for containers, as listed here.
This seems to be happening for me and my team at work, with latest versions of the Docker nginx image:
I've opened issues with them on the docker/compose GitHub here
this error appeared to me because my php-fpm image enabled cron, and I have no idea why
In my case it was nginx: [emerg] host not found in upstream as well, so I managed to solve it, by adding depends_on directive to nginx service in docker-compose.yml file.
(new to nginx)
In my case it was wrong folder name
For config
upstream serv {
server ex2_app_1:3000;
}
make sure the app folder is in ex2 folder:
ex2/app/...