Dynamic proxy_pass in nginx to another pod in Kubernetes - nginx

Question

I'm trying to create an nginx proxy that forwards requests to /<service> to http://<service>. I first tried the following:
location ~ ^/(.+)$ {
set $backend "http://$1:80";
proxy_pass $backend;
}
But it fails saying something like (when calling /myservice):
[error] 7741#0: *1 no resolver defined to resolve http://myservice
Since myservice is not externally accessible I've tried to install go-dnsmasq as a sidecar in the same pod and I try to use it for DNS resolution (like I've seen in this example) and change my nginx config to look like this:
location ~ ^/(.+)$ {
resolver 127.0.0.1:53;
set $backend "http://$1:80";
proxy_pass $backend;
}
But now nginx fails with:
[error] 9#9: *734 myservice could not be resolved (2: Server failure), client: 127.0.0.1, server: nginx-proxy, request: "GET /myservice HTTP/1.1", host: "localhost:8080"
127.0.0.1 - xxx [30/May/2016:10:34:23 +0000] "GET /myservice HTTP/1.1" 502 173 "-" "curl/7.38.0" "-"
My Kubernetes pod looks like this:
spec:
containers:
- name: nginx
image: "nginx:1.10.0"
ports:
- containerPort: 8080
name: "external"
protocol: "TCP"
- name: dnsmasq
image: "janeczku/go-dnsmasq:release-1.0.5"
args:
- --listen
- "0.0.0.0:53"
Running netstat -ntlp in the dnsmasq container gives me:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN -
tcp 0 0 :::53 :::* LISTEN 1/go-dnsmasq
And running nmap --min-parallelism 100 -sT -sU localhost in the nginx container:
Starting Nmap 6.47 ( http://nmap.org ) at 2016-05-30 10:33 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00055s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 1997 closed ports
PORT STATE SERVICE
53/tcp open domain
8080/tcp open http-proxy
53/udp open domain
So it seems that dnsmasq and nginx are indeed up and running? What could I be doing wrong?

Answer 1

After much research and trial and error I managed to solve this. First I changed the pod specification to:
spec:
containers:
- name: nginx
image: "nginx:1.10.0"
ports:
- containerPort: 8080
name: "external"
protocol: "TCP"
- name: dnsmasq
image: "janeczku/go-dnsmasq:release-1.0.5"
args:
- --listen
- "127.0.0.1:53"
- --default-resolver
- --append-search-domains
- --hostsfile=/etc/hosts
- --verbose
then I also had to disable the ipv6 for the resolver in nginx:
location ~ ^/(.+)$ {
resolver 127.0.0.1:53 ipv6=off;
set $backend "http://$1:80";
proxy_pass $backend;
}
Then it works as expected!

Answer 2

I resolved this by coredns docker ：
my nginx and coredns are all deploy on host
step1: config Corefile
in Corefile maybe you should change k8s master config refer: https://coredns.io/plugins/kubernetes/
sudo mkdir /etc/coredns; sudo tee /etc/coredns/Corefile <<-'EOF' .:53 {
log
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
endpoint http://172.31.88.71:8080
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance } EOF
step2:config docker and then start it
tee coreos.sh <<-'EOF'
docker run --restart=always -idt --name coredns \
-v /etc/coredns/Corefile:/etc/coredns/Corefile \
-v /home/ec2-user/.kube/config:/etc/coredns/kubeconfig \
-p 53:53/udp \
coredns/coredns:1.6.9 \
-conf /etc/coredns/Corefile
EOF
step3: config nginx and then reload
resolver 127.0.0.1 valid=60s ipv6=off;