How to redirect API calls with nginx? - nginx

Using nginx I am trying to redirect API calls to an external API provider website but after a lot of research and tries I feel I miss something important.
For example, my goal is when I open https://mywebsite/api/somedata, then a remote API provider is called https://api-somewebsite.com/somedata.
My basic try was:
location /api/ {
proxy_pass https://api-somewebsite.com/;
}
I added a log format to try to get more informations:
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name $host to: $upstream_addr: $request $status upstream_response_time $upstream_response_time msec $msec request_time $request_time';
Then used it to log the calls:
location /api/ {
proxy_pass https://api-somewebsite.com/;
access_log /var/log/nginx/upstream.log upstreamlog;
}
And it seems that the /api/ part is not well removed:
[21/Sep/2022:11:23:17 -0400] 88.163.105.196 - - - mywebsite.com mywebsite.com to: xxx.xx.xx.xx:xxx, xxx.xx.xx.xx:xxx: GET /api/somedata HTTP/1.1 502 upstream_response_time 0.005, 0.008 msec 1663773797.354 request_time 0.012
So basically it seems to call https://api-somewebsite.com/àpi/somedata which does not exist and thus return 502 error.
I have read that with the trailing / the /api/ part should be well removed automatically. But it does not work. So I tried to rewrite it by myself:
location /api/ {
proxy_pass https://api-somewebsite.com/;
access_log /var/log/nginx/upstream.log upstreamlog;
rewrite ^/api/(.*) /$1 break;
}
Still no luck.
I ended up trying some stuff found on the web but that I do not fully understand... like adding proxy_redirect instruction:
location /api/ {
proxy_pass https://api-somewebsite.com/;
access_log /var/log/nginx/upstream.log upstreamlog;
rewrite ^/api/(.*) /$1 break;
proxy_redirect https://api-somewebsite.com/ /api/;
}
Well, I feel that I lack some basic stuff to make it work, but I cannot find what. Any insights would be greatly appreciated.
====== EDIT 1 ======
According to Stephen Dunne answer, i tried to add an upstream for the server.
Here is my edited code:
upstream magiceden {
server api-mainnet.magiceden.dev:443 max_fails=0;
zone magiceden-api 64k;
keepalive 60;
}
server {
server_name genoverse.me www.genoverse.me;
location / {
root /home/hcomere/genoverse.me;
index index.html;
}
location /magiceden_api/ {
proxy_pass https://magiceden/;
access_log /var/log/nginx/access_magiceden.log upstreamlog;
}
listen [::]:443 ssl; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/genoverse.me/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/genoverse.me/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
I still have error 502 - Bad Gateway - when visiting https://www.genoverse.me/magiceden_api/
where i should get an error 400 - Not found - from the api provider
https://api-mainnet.magiceden.dev/
====== EDIT 2 ======
Oh well, i discovered that browsers have a redirect cache, this explain why sometimes things did not work where they should work.
Once i cleared the redirect cache after each config modification, things were way more smooth and understandable !...

I think the issue here is that you are using a Uri and not an upstream.
I have read that with the trailing / the /api/ part should be well removed automatically.
This is correct.
Try adding an upstream for the server.
upstream somesite {
server api-somewebsite.com:443 max_fails=0;
zone somesite-api 64k;
keepalive 60;
}
And then use the upstream in your location block (keeping the trailing slash)
location /api/ {
proxy_pass https://somesite/;
access_log /var/log/nginx/upstream.log upstreamlog;
}
Now when you browse to the /api location you should be redirected as expected.

Related

Cached version of the endpoint in nginx

I would like to have alternative urls for my API endpoints that would use nginx caching. For example a url /cached/api/songs/latest should return a cached version of the endpoint /api/songs/latest coming from the PHP engine.
I came up with this configuration but it does not work. Could you please advise what am I doing wrong?
proxy_cache_path /data/cache/nginx keys_zone=one:10m loader_threshold=300
loader_files=200 max_size=200m;
server {
server_name api.myserver.com;
# SSL configuration
listen 443 ssl;
ssl_certificate ...
ssl_certificate_key ...
root /data/myapp/public;
index index.html index.php;
# cached version of the endpoints
location ~ ^/cached/.*$ {
rewrite ^/cached(.*)$ /$1 break;
proxy_pass https://127.0.0.1:443;
proxy_cache_valid any 60m;
}
# standard includes
include global/restrictions.conf;
include global/common_php.conf;
include global/static_assets.conf;
}

How do I prevent nginx from redirecting to HTTPS in this particular setup?

I have a somewhat messy setup (no choice) where a local computer is made available to the internet through port forwarding. It is only reachable through [public IP]:8000. I cannot get a Let's Encrypt certificate for an IP address, but the part of the app that will be accessed from the internet does not require encryption. So instead, I'm planning on making the app available from the internet at http://[public IP]:8000/, and from the local network at https://[local DNS name]/ (port 80). The certificate used in the latter is issued by our network's root CA. Clients within the network trust this CA.
Furthermore, some small changes are made to the layout of the page when accessed from the internet. These changes are made by setting an embedded query param.
In summary, I need:
+--------------------------+--------------------------+----------+--------------------------------------+
| Accessed using | Redirect to (ideally) | URL args | Current state |
+--------------------------+--------------------------+----------+--------------------------------------+
| http://a.b.c.d:8000 | no redirect | embedded | Arg not appended, redirects to HTTPS |
| http://localhost:8000 | no redirect | embedded | Arg not appended, redirects to HTTPS |
| http://[local DNS name] | https://[local DNS name] | no args | Working as expected |
| https://[local DNS name] | no redirect | no args | Working as expected |
+--------------------------+--------------------------+----------+--------------------------------------+
For the two top rows, I don't want the redirection to HTTPS, and I need ?embedded to be appended to the URL.
Here's my config:
upstream channels-backend {
server api:5000;
}
# Connections from the internet (no HTTPS)
server {
listen 8000;
listen [::]:8000;
server_name [PUBLIC IP ADDRESS] localhost;
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /admin/ {
# Do not allow access to /admin/ from the internet.
return 404;
}
location /static/rest_framework/ {
alias /home/docker/backend/static/rest_framework/;
}
location /static/admin/ {
alias /home/docker/backend/static/admin/;
}
location /files/media/ {
alias /home/docker/backend/media/;
}
location /api/ {
proxy_pass http://channels-backend/;
}
location ~* (service-worker\.js)$ {
add_header 'Cache-Control' 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
expires off;
proxy_no_cache 1;
}
location / {
root /var/www/frontend/;
# I want to add "?embedded" to the URL if accessed through http://[public IP]:8000.
# I do not want to redirect to HTTPS.
try_files $uri $uri/ /$uri.html?embedded =404;
}
}
# Upgrade requests from local network to HTTPS
server {
listen 80;
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
server_name [local DNS name] [local IP] localhost;
# This works; it redirects to HTTPS.
return 301 https://$http_host$request_uri;
}
# Server for connections from the local network (uses HTTPS)
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name [local DNS name] [local IP] localhost;
ssl_password_file /etc/nginx/certificates/global.pass;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_protocols TLSv1.2 TLSv1.1;
ssl_certificate /etc/nginx/certificates/certificate.crt;
ssl_certificate_key /etc/nginx/certificates/privatekey.key;
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /admin/ {
proxy_pass http://channels-backend/admin/;
}
location /static/rest_framework/ {
alias /home/docker/backend/static/rest_framework/;
}
location /static/admin/ {
alias /home/docker/backend/static/admin/;
}
location /files/media/ {
alias /home/docker/backend/media/;
}
location /api/ {
# Proxy to backend
proxy_read_timeout 30;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $server_name;
proxy_redirect off;
proxy_pass http://channels-backend/;
}
# ignore cache frontend
location ~* (service-worker\.js)$ {
add_header 'Cache-Control' 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
expires off;
proxy_no_cache 1;
}
location / {
root /var/www/frontend/;
# Do not add "?embedded" argument.
try_files $uri $uri/ /$uri.html =404;
}
}
The server serves both the frontend and an API developed using React and Django RF, in case it matters. It's deployed using Docker.
Any pointers would be greatly appreciated.
Edit: I commented out everything except the first server (port 8000), and requests are still being redirected to https://localhost:8000 from http://localhost:8000. I don't understand why. I'm using an incognito tab to rule out cache as the problem.
Edit 2: I noticed that Firefox sets an Upgrade-Insecure-Requests header with the initial request to http://localhost:8000. How can I ignore this header and not upgrade insecure requests? This request was made by Firefox, and not the frontend application.
Edit 3: Please take a look at the below configuration, which I'm now using to try to figure out the issue. How can this possibly result in redirection from HTTP to HTTPS? There's now only one server block, and there's nothing here that could be interpreted as a wish to redirect to https://localhost:8000 from http://localhost:8000. Where does the redirect come from? Notice that I replaced some parts with redirects to Google, Yahoo and Facebook. I'm not redirected to any of these. I'm immediately upgraded to HTTPS, which should not be supported at all with this configuration. It's worth mentioning that the redirect ends in SSL_ERROR_RX_RECORD_TOO_LONG. The certificate is accepted when accessing https://localhost/ (port 80) using the original configuration.
upstream channels-backend {
server api:5000;
}
# Server for connections from the internet (does not use HTTPS)
server {
listen 8000;
listen [::]:8000 default_server;
server_name localhost [public IP];
keepalive_timeout 70;
access_log /var/log/nginx/access.log;
underscores_in_headers on;
ssl off;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /admin/ {
# Do not allow access to /admin/ from the internet.
return 404;
}
location /static/rest_framework/ {
alias /home/docker/backend/static/rest_framework/;
}
location /static/admin/ {
alias /home/docker/backend/static/admin/;
}
location /files/media/ {
alias /home/docker/backend/media/;
}
location /api/ {
proxy_pass http://channels-backend/;
}
location / {
if ($args != "embedded") {
return 301 https://google.com;
# return 301 http://$http_host$request_uri?embedded;
}
return 301 https://yahoo.com;
# root /var/www/frontend/;
# try_files $uri $uri/ /$uri.html =404;
}
}
Boy, do I feel stupid.
In my docker-compose.yml file, I had accidentally mapped port 8000 to 80:
nginx-server:
image: nginx-server
build:
context: ./
dockerfile: .docker/dockerfiles/NginxDockerfile
restart: on-failure
ports:
- "0.0.0.0:80:80"
- "0.0.0.0:443:443"
- "0.0.0.0:8000:80" # Oops
So any request on port 8000 was received by nginx as a request on port 80. Thus, even a simple config like...
server {
listen 8000;
return 301 https://google.com;
}
... would result in an attempt to upgrade to HTTPS (causes include unexpected caching of redirects, possibly default behavior, etc.) on port 80. I was thoroughly confused, but fixing my compose instructions fixed the problem:
nginx-server:
image: nginx-server
build:
context: ./
dockerfile: .docker/dockerfiles/NginxDockerfile
restart: on-failure
ports:
- "0.0.0.0:80:80"
- "0.0.0.0:443:443"
- "0.0.0.0:8000:8000" # Fixed

Nginx reverse proxy doesn't work as expected

I have two domains on different nginx(1.15.0) servers (server1 example.com and server2 example.net). I've tried to set up server2 as a reverse proxy with ngx_http_substitutions_filter_module but it doesn't work as expected.
Due to my config, subs_filter directive should replace example.com to example.net but when I type example.net in browser it redirects me to example.com.
nginx.conf
http {
//other settings
.....
include /etc/nginx/sites-enabled/*;
upstream example.com {
server example.com;
}
}
example.net.conf
server {
listen 80;
server_name example.net www.example.net;
rewrite ^/(.*)$ https://example.net/$1 permanent;
}
server {
listen 443 ssl;
server_name example.net;
ssl_certificate /etc/letsencrypt/live/example.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.net/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
root html;
try_files $uri #example.com;
}
location #example.com {
include replace.conf;
proxy_pass http://example.com;
proxy_cookie_domain example.com example.net;
proxy_set_header Accept-Encoding "";
proxy_set_header Host example.com;
proxy_redirect http://example.com http://example.net;
}
}
replace.conf
# replace direct links
subs_filter "www.example.com" "example.net" gi;
subs_filter "example.com" "example.net" gi;
Seems like nginx ignores subs_filter directive.
Could someone explain me how can I replace uri properly using ngx_http_substitutions_filter_module? Thank you for advice!
Problem solved.
First of all I removed upstream from nginx.conf:
upstream example.com {
server example.com;
}
Then I changed following lines in example.net.conf:
location / {
...
try_files $uri #static;
}
location #static {
...
proxy_pass https://example.com;
...
...
...
proxy_redirect https://example.com https://example.net;
}
All works fine except login form. Firefox works correctly but Chrome returns error 422. I suppose this is because the login form works on javascript.
Anyway thanks to all!

Authorization (401) issue with Nginx

I have a Nginx virtualhost with the following content:
server {
listen 80;
location /.well-known {
alias /usr/local/etc/letsencrypt-webroot/.well-known;
}
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/sub.domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sub.domain.com/privkey.pem;
ssl_dhparam /etc/nginx/ssl/dh.pem;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
location / {
proxy_pass http://127.0.0.1:3000;
proxy_cookie_path / "/; secure; HttpOnly";
auth_basic "No no no!";
auth_basic_user_file /etc/nginx/.htpasswd;
}
}
I'm having a problem when the URL is /.well-known/.... I get a 401 because it keeps asking for credentials.
So my questions are:
Why is /.well-known user-protected? Shouldn't it be processed before reaching the location / in the SSL section, which is the one requiring authentication?
How can I fix this? Basically I need that everything requested to the proxy needs authentication except for /.well-known.
location /.well-known {
auth_basic off;
}
Try opening http://yourserver/.well-known/ instead of https://yourserver/.well-known.
If this works reconsider moving both location blocks to single server {...} block.
//edit to make it working you have to either remove redirect from. well-known block or adding it to block in https server definition.

nginx rewrite not working

I'm trying to set up a simple nginx server to act as a proxy between my front end ui and my back end api. The setup is fairly simple. The UI makes all api requests to /api/endpoint and the proxy server passes the request to the api. The proxy also needs to rewrite the request so that instead of going to http://api.location.net/api/endpoint, it goes to http://api.location.net/endpoint. The UI resides on http://api.location.net. This part isn't working (i get a 500 error) and I'm pretty sure it has to do with how I'm writing my rewrite rule. Here's my nginx config.
daemon off;
error_log off;
worker_processes 2;
worker_rlimit_nofile 100000;
events {
worker_connections 50000;
accept_mutex off;
}
http {
include /etc/nginx/mime.types;
access_log off;
sendfile on;
server {
listen 80 default_server;
server_name localhost _;
location / {
alias /srv/site/;
}
location /api/ {
rewrite ^/api ""; # I think this is the problem
proxy_pass http://api.location.net;
proxy_pass_request_headers on;
proxy_pass_header X-ResponseData;
proxy_redirect off;
}
}
}
Any help would be greatly appreciated, nginx is still fairly new for me and the documentation on nginx rewrite doesn't seem to have what I need.
If I understood you right, this should help
location /api/ {
proxy_pass http://api.location.net/;
proxy_pass_request_headers on;
proxy_pass_header X-ResponseData;
proxy_redirect off;
}
Note the URI part at proxy_pass directive
If the proxy_pass directive is specified with a URI, then when a
request is passed to the server, the part of a normalized request URI
matching the location is replaced by a URI specified in the directive:
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

Resources