I have 3 ( app-client,app-a, app-b )applications are running in jetty server and one NGINX load balancer ( app-lb). All (internal or external) requests are coming to my application through load balancer. Based on web context ( /app-a/ or /app-b/) name, LB will forward the request to the correct application. I have configured (location /app-a/ and location /app-b and location /app-client) in LB. app-a will call app-b and app-b will call app-a , app-client will be called from outside world, app-client will call app-a or app-b.
I have written Docker-composer for my application. To avoid circular dependency, I have used Docker net. it is working good.
If I scale up my application. LB doesn't know about this new application container.
I have gone though few tutorials and trying to use jwilder/nginx-proxy instead of NGINX. if I use that using VIRTUAL_HOST=app-name variable it is updating upstream in configuration file.But, My application is running based on location mapping for each container.if I don't specify, how request will go to correct container.
How to give location mapping in LB's default.conf file like below since this configuration updated by container dynamically or how to make internal call urls.
location /app-a {
proxy_pass http://app-a;
}
location /app-client {
proxy_pass http://app-client;
}
location /app-b {
proxy_pass http://app-b;
}
Request from outside: http://IP:9090/app-client/
Internal call : http://app-lb:80/app-a/
http://app-lb:80/app-b
LB exposed port no is 9090
There are pull requests (e.g. #599) for the nginx-proxy image to support virtual paths. To implement this, you can use the original image and just pass your own nginx.tmpl file into the container (as a volume mount, e.g. -v $(pwd)/nginx.tmpl:/app/nginx.tmpl:ro). Then your containers just need to define VIRTUAL_PATH as they would VIRTUAL_HOST.
I'd also recommend setting DEFAULT_HOST on the nginx-proxy container and have everyone point to that if you don't want hostname based routing.
Note with #599, there's a bug in the nginx.tmpl that I ran into, you need to move {{ $networks := .Networks }} up two lines to be before the {{ range $container := .Containers }} (the range redefines . which redefines .Networks). Otherwise all networks will be assumed reachable and you'll get timeouts if the container is also attached to other networks that nginx-proxy can't reach.
Related
I have a single application running in multiple K8s clusters; Let's say there is a frontend service, and two backend ones.
I use NGINX proxy the requests from the frontend to the backend services. Regular NGINX edition, not NGINX+.
Here is the nginx.conf:
server {
....
set $back1 "<k8s hostname for the backend1 service>";
set $back2 "<k8s hostname for the backend2 service>";
location /back1 {
rewrite ^/back1/(.*)$ /$1 break;
proxy_pass http://$back1;
}
<and same for the backend 2 service>
}
So basically, what happens is that in my frontend application, I set the backend service address to localhost/back1 and localhost/back2, the requests hit NGINX which strips off those back1 and back2 prefixes and call whatever endpoint I specify after in the actual backend services in K8s.
As I have multiple K8s clusters, the backend services hostnames differ, and I need to account for that in my NGINX conf.
The question is:
Is there a way for NGINX to differentiate between my K8s clusters?
Perhaps I can pass an environment variable to the container running my frontend service, and make an if statement in nginx.conf. Something like:
server {
if (${env} = "cluster1") {
set $back1 = "<cluster1 hostname>"
}
if (${env} = "cluster2") {
set $back1 = "<cluster2 hostname>"
}
}
Or if I can execute a shell command in the nginx conf to get the hostname and write similar if blocks.
I would appreciate any help on this matter!
I went a different route - via templates, environment variables, and envsubst utility which is shipping in the latest nginx docker images.
In template:
set $upstream_back1 "${BACK1}";
set $upstream_back2 "${BACK2}";
In Dockerfile
RUN envsubst < yourtemplate > /etc/nginx/nginx.conf
I am running nginx as part of the docker-compose template.
In nginx config I am referring to other services by their docker hostnames (e.g. backend, ui).
That works fine until I do that trick:
docker stop backend
docker stop ui
docker start ui
docker start backend
which makes backend and ui containers to exchange IP addresses (docker provides private network IPs on a basis of giving the next IP available in CIDR to each new requester). This 4 commands executed imitate some rare cases when both upstream containers got restarted at the same time but the nginx container did not. Also, I believe, this should be a very common situation when running pods on Kubernetes-based clusters.
Now nginx resolves backend host to ui's IP and ui to backend's IP.
Reloading nginx' configuration does help (nginx -s reload).
Also, if I do nslookup from within the nginx container - the IPs are always resolved correctly.
So this isolates the problem to be a pure nginx issue around the DNS caching.
The things I tried:
I have the resolver set under the http {} block in nginx config:
resolver 127.0.0.11 ipv6=off valid=10s;
Most common solution proposed by the folks on the internet to use variables in proxy-pass (this helps to prevent nginx to resolve and cache DNS records on start) - that did not make ANY difference at all:
server {
<...>
set $mybackend "backend:3000";
location /backend/ {
proxy_pass http://$mybackend;
}
}
Tried adding resolver line into the location itself
Tried setting the variable on the http{} block level, using map:
http {
map "" $mybackend {
default backend:3000;
}
server {
...
}
}
Tried to use openresty fork of nginx (https://hub.docker.com/r/openresty/openresty/) with resolver local=true
None of the solutions gave any effect at all. The DNS caches are only wiped if I reload nginx configuration inside of the container OR restart the container manually.
My current workaround is to use static docker network declared in docker-compose.yml. But this has its cons too.
Nginx version used: 1.20.0 (latest as of now)
Openresty versions used: 1.13.6.1 and 1.19.3.1 (latest as of now)
Would appreciate any thoughts
UPDATE 2021-09-08: Few months later I am back to solving this same issue and still no luck. Really looks like the bug in nginx - I can not make nginx to re-resolve the dns names. There seems to be no timeout to nginx' dns cache and none of the options listed above to introduce timeouts or trigger dns flush work.
UPDATE 2022-01-11: I think the problem is really in the nginx. I tested my config in many ways a couple months ago and it looks like something else in my nginx.conf prevents the valid parameter of the resolver directive from working properly. It is either the limit_req_zone or the proxy_cache_path directives used for request rate limiting and caching respectively. These just don't play nicely with the valid param for some reason. And I could not find any information about this anywhere in nginx docs.
I will get back to this later to confirm my hypothesis.
Maybe it's because nginx's DNS resolver for upstream servers only works in the commercial version, nginx plus?
https://www.nginx.com/products/nginx/load-balancing/#service-discovery
TLDR: Your Internet Provider may be caching dnses with no respect to tiny TTL values (like 1 second).
I've been trying to retest locally the same thing.
Your docker might be using local resolver (127.0.0.11)
Then Dns might be cached by your OS (which you may clean - that's OS specific)
Then you might have it cached on your WIFI/router (yes!)
Later it goes to your ISP and is beyond your control.
But nslookup is your friend, you can query each dns server between nginx and root DNS server.
Something very easy to reproduce (without setting up local dns server)
Create route 53 'A' entry with TTL of 1 second and try to query AWS dns server in your hosted zone (it will be sth. like ns-239.awsdns-29.com)
Play around with dig / nslookup command
nslookup
set type=a
server ns-239.awsdns-29.com
your.domain.com
It will return IP you have set
Change the Route53 'A' entry to some other ip.
use dig / nslookup and make sure you see changes immediately
Then set resolver in nginx to AWS dns name (for testing purposes only).
If that works it means that DNS is cached elsewere and this is no longer nginx issue!
In my case it was sunrise WIFI router which began to see new IP only after I restarted it (I assume things would resolve after some longer value).
Great help when debugging this is when your nginx is compiled with
--with-debug
Then in nginx logs you see whether given dns was resolved and to what IP.
My whole config looks like this (here with standard docker resolver which has to be set if you are using variables in proxy_pass!)
server {
listen 0.0.0.0:8888;
server_name nginx.my.custom.domain.in.aws;
resolver 127.0.0.11 valid=1s;
location / {
proxy_ssl_server_name on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Host $host;
set $backend_servers my.custom.domain.in.aws;
proxy_pass https://$backend_servers$request_uri;
}
}
Then you can try to test it with
curl -L http://nginx.my.custom.domain.in.aws --resolve nginx.my.custom.domain.in.aws 0.0.0.0:8888
Was struggling on the same thing exactly for the same thing (Docker Swarm) and actually to make it work I required to let the upstream away from my configuration.
Something that works well (tested 5' ago on NGINX 2.22) :
location ~* /api/parameters/(.*)$ {
resolver 127.0.0.11 ipv6=off valid = 1s;
set $bck_parameters parameters:8000;
proxy_pass http://$bck_parameters/api/$1$is_args$args;
}
where $bck_parameters is NOT an upstream but the real server behind.
Doing same thing with upstream will fail.
I'm attempting to re-run a state from another state. I'm not using watch or watch_in etc b/c i want it to run each time. I configure all my nginx virtual hosts and then at the end another state runs called nginx-certs the relevant portion is here:
nginx-frontend:
module.run:
- name: state.sls
- mods:
- nginx-frontend
During the highstate i see the state_id is executed but has no comments, nor shows it reruns that state, it just completes as Result: True. I can then jump to the salt master and run
sudo salt webserver state.sls nginx-certs
and when it hits nginx-frontend, it does reload all of the virtual hosts, putting the new cert in the config.
I'm curious why this does not run in the highstate.
I have attempted ll sorts of different variations of the simple block outlined above. This one works, but not in the highstate, which is my goal to fix.
If you wonder why i do it this way, all certificates for production and staging terminate at HAProxy and nginx only serves up 80/http1 81/h2, but when building out dev servers i want to assign the cert directly to the server as it will be public facing. I need to build out the virtual hosts first to get port 80 open which is used for lets-encrypt. Then when the cert is available, update the nginx vhosts listen directive and cert paths.
From what I understand: you have one server which you want temporarily configured with Nginx on port 80, then generate its certificate with letsencrypt, then change Nginx configuration to be on port 443.
What you can do is:
have one state which installs and configures Nginx to listen on port 80
have another state with installs/configures/runs letsencrypt
a third state which configures Nginx as you want it to be at the end [1]
you just include them in salt to be run in the specific order like
# custom_nginx.sls
include:
- temp_nginx_on_port_80
- letsencrypt_cert
- nginx
[1] for this I think its better to use formula like the one from the community https://github.com/saltstack-formulas/nginx-formula/ and configure it with pillar data. Obviously if you use it for step 3, you won't be able to use for step 1 (or at least I don't see right now how)
I'm successfully able to run a flask app on my IP:5000 path. A simple Hello World program that shows the output on my browser.
Now, what I would like to do is to configure NGINX with a proxy so that if I access only IP which apparently runs on a default port 80, it should navigate to port 5000 and show output of my application.
In other words...
This is working : IP:5000 -> Output = Hello world
This isn't working: IP -> This site can’t be reached
The server settings that I want to add would be something like this.
server {
listen 80;
server_name MY_IP;
location / {
proxy_pass http://127.0.0.1:5000;
}
}
However, I'm not sure where to add this? Should it be inside http block inside /etc/nginx/nginx.conf?
Updates: Based on the answers given below, I've managed to do the following.
I did restart nginx after this. However, I'm still facing the same issue. App works on IP:5000 but does not work on IP
The configuration you have mentioned should be in a separate file, assume example.com.conf under /etc/nginx/conf.d. You can put all the configuration in /etc/nginx/nginx.conf and it'll work, it's just that for readability we create separate configuration files which would be auto included when you add it inside conf.d.
Ok, the problem is fixed. As #senaps and #Mukanahallipatna had mentioned, I created the new configuration file under conf.d.
However, the most imp step that I was missing was this part mentioned in the below link.
It is recommended that you enable the most restrictive profile that will still allow the traffic you've configured. Since we haven't configured SSL for our server yet, in this guide, we will only need to allow traffic on port 80.
Reference Link
sudo ufw allow 'Nginx HTTP'
Now, everything is working fine.
Put the working blocks in a file with any_name.conf inside the folder named /etc/nginx/conf.d and it will be loaded automatically.
You will need to restart your nginx.
update:
What are you using to serve flask? if you are using uwsgi, then you should use configurations like this:
include uwsgi_params;
uwsgi_pass unix:path_to_your.sock;
Other options for uwsgi_pass are:
uwsgi_pass localhost:9000; #normal
uwsgi_pass uwsgi://localhost:9000;
uwsgi_pass suwsgi://[2001:db8::1]:9090; #uwsgi over ssl
If you are using gunicorn to serve your flask app, then your current configs should be fine, check if your app is running and if you can get your index page or not using 5000 port, then check for other problems. your configs looks good, maybe it's a problem on flask not being run?
I use nginx to proxy and hold persistent connections to far away servers for me.
I have configured about 15 blocks similar to this example:
upstream rinu-test {
server test.rinu.test:443;
keepalive 20;
}
server {
listen 80;
server_name test.rinu.test;
location / {
proxy_pass https://rinu-test;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
}
}
The problem is if the hostname can not be resolved in one or more of the upstream blocks, nginx will not (re)start. I can't use static IPs either, some of these hosts explicitly said not to do that because IPs will change. Every other solution I've seen to this error message says to get rid of upstream and do everything in the location block. That it not possible here because keepalive is only available under upstream.
I can temporarily afford to lose one server but not all 15.
Edit:
Turns out nginx is not suitable for this use case. An alternative backend (upstream) keepalive proxy should be used. A custom Node.js alternative is in my answer. So far I haven't found any other alternatives that actually work.
Earlier versions of nginx (before 1.1.4), which already powered a huge number of the most visited websites worldwide (and some still do even nowdays, if the server headers are to be believed), didn't even support keepalive on the upstream side, because there is very little benefit for doing so in the datacentre setting, unless you have a non-trivial latency between your various hosts; see https://serverfault.com/a/883019/110020 for some explanation.
Basically, unless you know you specifically need keepalive between your upstream and front-end, chances are it's only making your architecture less resilient and worse-off.
(Note that your current solution is also wrong because a change in the IP address will likewise go undetected, because you're doing hostname resolution at config reload only; so, even if nginx does start, it'll basically stop working once IP addresses of the upstream servers do change.)
Potential solutions, pick one:
The best solution would seem to just get rid of upstream keepalive as likely unnecessary in a datacentre environment, and use variables with proxy_pass for up-to-date DNS resolution for each request (nginx is still smart-enough to still do the caching of such resolutions)
Another option would be to get a paid version of nginx through a commercial subscription, which has a resolve parameter for the server directive within the upstream context.
Finally, another thing to try might be to use a set variable and/or a map to specify the servers within upstream; this is neither confirmed nor denied to have been implemented; e.g., it may or may not work.
Your scenario is very similar to the one when using aws ELB as uptreams in where is critical to resolve the proper IP of the defined domain.
The first thing you need to do and ensure is that the DNS servers you are using can resolve to your domains, then you could create your config like this:
resolver 10.0.0.2 valid=300s;
resolver_timeout 10s;
location /foo {
set $foo_backend_servers foo_backends.example.com;
proxy_pass http://$foo_backend_servers;
}
location /bar {
set $bar_backend_servers bar_backends.example.com;
proxy_pass http://$bar_backend_servers;
}
Notice the resolver 10.0.0.2 it should be IP of the DNS server that works and answer your queries, depending on your setup this could be a local cache service like unbound. and then just use resolve 127.0.0.1
Now, is very important to use a variable to specify the domain name, from the docs:
When you use a variable to specify the domain name in the proxy_pass directive, NGINX re‑resolves the domain name when its TTL expires.
You could check your resolver by using tools like dig for example:
$ dig +short stackoverflow.com
In case is a must to use keepalive in the upstreams, and if is not an option to use Nginx +, then you could give a try to openresty balancer, you will need to use/implement lua-resty-dns
A one possible solution is to involve a local DNS cache. It can be a local DNS server like Bind or Dnsmasq (with some crafty configuration, note that nginx can also use specified dns server in place of the system default), or just maintaining the cache in hosts file.
It seems that using hosts file with some scripting is quite straightforward way. The hosts file should be spitted into the static and dynamic parts (i.e. cat hosts.static hosts.dynamic > hosts), and the dynamic part should be generated (and updated) automatically by a script.
Perhaps it make sense to check from time to time the hostnames for changing IPs, and update hosts file and reload configuration in nginx on changes. In case of some hostname cannot be resolved the old IP or some default IP (like 127.0.1.9) should be used.
If you don't need the hostnames in the nginx config file (i.e., IPs are enough), the upstream section with IPs (resolved hostnames) can be generated by a script and included into nginx config — and no need to touch the hosts file in such case.
I put the resolve parameter on server and you need to set the Nginx Resolver in nginx.conf as below:
/etc/nginx/nginx.conf:
http {
resolver 192.168.0.2 ipv6=off valid=40s; # The DNS IP server
}
Site.conf:
upstream rinu-test {
server test.rinu.test:443;
keepalive 20;
}
My problem was container related. I'm using docker compose to create the nginx container, plus the app container. When setting network_mode: host in the app container config in docker-compose.yml, nginx was unable to find the upstream app container. Removing this fixed the problem.
we can resolve it temporarily
cd /etc
sudo vim resolv.conf
i
nameserver 8.8.8.8
:wq
then do sudo nginx -t
restart nginx it will work for the momment
An alternative is to write a new service that only does what I want. The following replaces nginx for proxying https connections using Node.js
const http = require('http');
const https = require('https');
const httpsKeepAliveAgent = new https.Agent({ keepAlive: true });
http.createServer(onRequest).listen(3000);
function onRequest(client_req, client_res) {
https.pipe(
protocol.request({
host: client_req.headers.host,
port: 443,
path: client_req.url,
method: client_req.method,
headers: client_req.headers,
agent: httpsKeepAliveAgent
}, (res) => {
res.pipe(client_res);
}).on('error', (e) => {
client_res.end();
})
);
}
Example usage:
curl http://localhost:3000/request_uri -H "Host: test.rinu.test"
which is equivalent to:
curl https://test.rinu.test/request_uri