NGINX & Consul-Template in Docker - nginx

Question

I'm having trouble with consistent service discovery using EC2, AWS, Docker, Consul-Template, Consul, and NGINX.
I have multiple services, each running on it's own EC2 instance. On these instances I run the following containers (in this order):
cAdvisor (monitoring)
node-exporter (monitoring)
Consul (running in agent mode)
Registrator
My service
Custom container running both nginx and consul-template
The custom container has the following Dockerfile:
FROM nginx:1.9
#Install Curl
RUN apt-get update -qq && apt-get -y install curl
#Install Consul Template
RUN curl -L https://github.com/hashicorp/consul-template/releases/download/v0.10.0/consul-template_0.10.0_linux_amd64.tar.gz | tar -C /usr/local/bin --strip-components 1 -zxf -
#Setup Consul Template Files
RUN mkdir /etc/consul-templates
COPY ./app.conf.tmpl /etc/consul-templates/app.conf
# Remove all other conf files from nginx
RUN rm /etc/nginx/conf.d/*
#Default Variables
ENV CONSUL consul:8500
CMD /usr/sbin/nginx -c /etc/nginx/nginx.conf && consul-template -consul=$CONSUL -template "/etc/consul-templates/app.conf:/etc/nginx/conf.d/app.conf:/usr/sbin/nginx -s reload"
The app.conf file looks like this:
{{range services}}
upstream {{.Name}} {
least_conn;{{range service .Name}}
server {{.Address}}:{{.Port}};{{end}}
}
{{end}}
server {
listen 80 default_server;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
location / {
proxy_pass http://cart/cart/;
}
location /cart {
proxy_pass http://cart/cart;
}
{{range services}}
location /api/{{.Name}} {
proxy_read_timeout 180;
proxy_pass http://{{.Name}}/{{.Name}};
}
{{end}}
}
Everything seems to start up perfectly ok, but at some point (which I'm yet to identify) after start up, consul-template seems to return that there are no available servers for a particular service. This means that the upstream section for that service contains no servers, and I end up with this in the logs:
2015/12/04 07:09:34 [emerg] 77#77: no servers are inside upstream in /etc/nginx/conf.d/app.conf:336
nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/app.conf:336
2015/12/04 07:09:34 [ERR] (runner) error running command: exit status 1
Consul Template returned errors:
1 error(s) occurred:
* exit status 1
2015/12/04 07:09:34 [DEBUG] (logging) setting up logging
2015/12/04 07:09:34 [DEBUG] (logging) config:
{
"name": "consul-template",
"level": "WARN",
"syslog": false,
"syslog_facility": "LOCAL0"
}
2015/12/04 07:09:34 [emerg] 7#7: no servers are inside upstream in /etc/nginx/conf.d/app.conf:336
nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/app.conf:336
After this, NGINX will no longer accept requests.
I'm sure I'm missing something obvious, but I've tied myself in mental knots about the sequence of events etc. What I think might be happening is that NGINX crashes, but because consul-template is still running, the Docker container doesn't restart. I don't actually care if the container itself restarts, or if just NGINX restarts.
Can someone help?

Answer 1

Consul Template will exit once the script it runs after writing returns a non-zero exit code. See here for the documentation.
The documentation suggests to put a || true just after the restart (or reload) command. This will keep Consul Template running independent of the exit code.
You could consider wrapping the restart in its own shell script that first tests the configuration (with nginx -t) before triggering a reload. You could even move the initial start of nginx to this script as it only makes sense to start nginx once the first (valid) configuration has been written?!