NGINX Reverse Proxy Fails with 404 despite being able to curl endpoint - nginx

I'm trying to reverse proxy to another endpoint on /alertmanager but it fails to connect. Weirdly enough I'm able to connect the endpoint directly from inside the pod running nginx.
A quick overview of my application architecture is this:
nginx ingress on cluster-> nginx load balancer -> <many services on different endpoints>
This is a minimized nginx configuration that replicates the issue:
worker_processes 5; ## Default: 1
error_log /dev/stderr;
pid /tmp/nginx.pid;
worker_rlimit_nofile 8192;
events {}
http {
client_body_temp_path /tmp/client_temp;
proxy_temp_path /tmp/proxy_temp_path;
fastcgi_temp_path /tmp/fastcgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
scgi_temp_path /tmp/scgi_temp;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] $status '
'"$request" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /dev/stderr main;
sendfile on;
tcp_nopush on;
resolver kube-dns.kube-system.svc.cluster.local;
server {
listen 8080;
proxy_set_header X-Scope-OrgID 0;
location = / {
return 200 'OK';
auth_basic off;
}
location /alertmanager {
proxy_pass http://mimir-distributed-alertmanager.mimir.svc.cluster.local:8080$request_uri;
}
}
}
I'm able to curl to the mimir endpoint in /alertmanager but I can't reach /alertmanager without getting a 404 error but I can get to / and if I put the proxy_pass inside of / it does work.
Example of what I'm seeing:
/ $ curl localhost:8080/
OK
/ $ curl localhost:8080/alertmanager
the Alertmanager is not configured
Curling http://mimir-distributed-alertmanager.mimir.svc.cluster.local does infact return the html of the page I'm expecting

Related

nginx is not proxy passing to my nest app

I am trying to configure NGINX to serve my nest app(which is running on docker).
My app is listening on port 3000
The server is amazon linux 2(ec2-user)
The conf file looks like this:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 4096;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
#include /etc/nginx/conf.d/nginx.conf;
server {
listen 80;
listen [::]:80;
server_name <ip.adress>;
#web
location / {
add_header X-yahav $uri; # this gets mounted
}
#api
location = /api { # this one is never approached
add_header X-yahav "Api-pass";
#proxy_set_header X-Real-IP $remote_addr;
#proxy_set_header Host $http_host;
#proxy_pass http://127.0.0.1:3000/;
}
}
}
I want to redirect /api to my nest app but it just wont have it i'm getting a simple 404 without the header i'm attaching(as you can see in the conf file)
Another thing is when I go to the root (location /) I do get my header mounted as expected
Any have any idea what is wrong?
Don't forget to add "/api" to your requests, like http://your-server:80/api/

'connection reset by peer' for large response body by nginx mirror module

I want to copy requests to another backend with ngx_http_mirror_module.
This is my nginx.conf.
Nginx version is 1.19.10
worker_processes 1;
error_log logs/error.log info;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main;
sendfile on;
keepalive_timeout 65;
server {
listen 8888;
server_name localhost;
location / {
proxy_pass http://127.0.0.1:8080;
mirror /mirror;
}
location /mirror {
internal;
proxy_pass http://127.0.0.1:18080$request_uri;
}
}
}
My Spring applications listen on 8080 and 18080.
The problem is that when the backend server which handles the mirrored request returns a large body response, the backend server throws ClientAbortException because of connection reset by peer.
Nothing is recorded in the nginx error log.
The nginx access log records status 200 for the mirrored request.
Problems tend to occur when the response size is 4k bytes or larger.
Increasing proxy_buffer_size may solve the problem, but if the response size is large (8k bytes or more?), Even if it is smaller than proxy_buffer_size, problems will occur.
I tried to change subrequest_output_buffer_size, but nothing changed.
How can I stop the error?

Nginx can't resolve server names when loading them from a file

I'm new to nginx and trying to achieve the following behaviour: I want to pass an header to nginx and pass the request to another server based on the given header. I want to load my map from a file so i'm using the following code
worker_processes 1;
error_log logs/error.log;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
sendfile on;
tcp_nopush on;
keepalive_timeout 65;
map $http_dest $new_server{
include port_to_address.map;
default google.com;
}
server {
listen 200;
server_name localhost;
access_log logs/host.access.log main;
location / {
proxy_pass http://$new_server;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
with the port_to_address.map in the following format
value_1 yahoo.com;
value_2 neflix.com;
With This configuration nginx starts normally but when i pass him an header that exists in the file he returns the following error
2021/04/18 21:52:49 [error] 17352#20504: *3 no resolver defined to resolve yahoo.com, client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost:200"
When i'm using the ip address and not the server's name it works perfectly. I also got another nginx thats round-robin requests between nodes it reads from a file (with upstream) and there i doesn't get any exception although im using the server's name and not their ip.
I read about using resolver but it doesn't work and i prefer to avoid it. Is there any other way to make it work without using the resolver or the servers ip? (I dont mind changing the map file structure if needed)

Testing load balancing in NGINX

I set up load balancing on NGINX using the Round Robin for apache tomcat servers with two servers in my proxy.conf file:
upstream appcluster1 {
server IP_ADDRESS:8000;
server IP_ADDRESS:8001;
}
server {
location / {
proxy_pass http://appcluster1;
}
}
This is deployed on the cloud and I am able to hit the endpoint using this method successfully. However, I want to test and see if nginx redirects between the two servers. How would I go about this?
I tried this method but I do not see anything in the logs that shows what server it is hitting. Is there any other way I can test and see if nginx would go to the second server?
EDIT: I have another file called nginx.conf that looks like this:
load_module modules/ngx_http_js_module.so;
user nginx;
worker_processes auto;
events {
worker_connections 2048;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
js_include auth.js;
proxy_buffering off;
log_format upstreamlog '$server_name to: $upstream_addr {$request} '
'upstream_response_time $upstream_response_time'
' request_time $request_time';
# log_format main '$remote_addr - $remote_user [$time_local] $status '
# '"$request" $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
# access_log logs/access.log main;
# sendfile on;
#tcp_nopush on;
keepalive_timeout 65s;
proxy_connect_timeout 120s;
keepalive_requests 50;
include /etc/nginx/conf.d/*.conf;
}

Nginx HLS Load balancer data traffic on primary server

My design is:
Media Server -> edge servers (Multiple Nginx Cache server -> Nginx Load Balancer)
it is my private CDN system (for Live content delivery)
I have content source and multiple edges; in each edge, there is multiple cache server and a Load Balancer
I started step by step, so for this job I face a problem with Nginx Load Balancer.
In this configuration, I am balancing between two servers s1 and s2.
but when I check traffic by(nload), I see big traffic on the primary server(load balancer) for example from nload see s1=1GBPS s2=1GBPS loadbalancer=2GBPS
note: my content is HLS (.m3u8)
user www-data;
worker_processes 5; ## Default: 1
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
worker_rlimit_nofile 8192;
events {
worker_connections 4096; ## Default: 1024
}
http {
include mime.types;
include /etc/nginx/proxy.conf;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] $status '
'"$request" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
# upstream
upstream origins {
server s1.ip;
server s2.ip;
}
# default route
server {
listen 80;
server_name example.com;
access_log /var/log/nginx/example.com main;
location / {
proxy_set_header Host $host;
proxy_pass http://origins;
}
}
}

Resources