Calling external api in Nginx location section - nginx

I am trying to resolve proxy_pass value dynamically (through web api) in nginx.
I need something like below;
Example taken from: https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/
location /proxy-pass-uri {
set $urlToProxy = CallWebAPI("http://localhost:8081/resolver?url=" + $url);
proxy_pass $urlToProxy;
}
So, my question is that, is it possible to make HTTP request or to write method such as CallWebAPI?
I know it might be a bad practice, but the website I am dealing with has thousands of web urls, which are mapped as key-value pairs, and 90% of them does not obey any specific regex rules. So I have content mapped database, and I need to fetch incoming url with content dynamically.
I am trying to use a very light web service to look up URLs from redis, and return proxy url.
Would this be a valid scenario, or is there any other built in solution in nginx like this?

I doubt this can be done with "pure" nginx, but this definitely can be done with openresty or ngx_http_lua_module with the help of ngx.location.capture method. For example:
resolver 8.8.8.8;
location ~/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("http://localhost:8081/resolver?url=" .. ngx.var.url)
ngx.var.proxy = res.body
}
proxy_pass $proxy$url;
}
There is also an ngx_http_js_module (documentation, GitHub) which have an ability to do subrequests (example), but I never used it and cannot tell if it can be used this way.
Important update
After almost a three years since this answer was written, it comes that I needed the similar functionality myself, and it turns out that the above answer is completely broken and unworkable. You can't do a subrequest via ngx.location.capture to anything else but to some other nginx location. So the correct (checked and confirmed to be workable) example for the above question is
resolver 8.8.8.8;
location /resolver {
internal;
proxy_pass http://localhost:8081;
}
location ~ ^/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("/resolver?url=" .. ngx.var.url)
if res.status == ngx.HTTP_OK then
ngx.var.proxy = res.body
else
ngx.exit(res.status)
end
}
proxy_pass $proxy$url$is_args$args;
}
The above example assumes that the proxy resolution service is really expecting request in a /resolver?url=<uri> form. The location /resolver { ... } while being internal behaves like any other prefix location, so if the /resolver prefix for that location cannot be used for some reason, the same can be written as
resolver 8.8.8.8;
location /get_proxy {
internal;
proxy_pass http://localhost:8081/resolver;
}
location ~ ^/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("/get_proxy?url=" .. ngx.var.url)
if res.status == ngx.HTTP_OK then
ngx.var.proxy = res.body
else
ngx.exit(res.status)
end
}
proxy_pass $proxy$url$is_args$args;
}

Related

nginx - Proxy to a different backend based on query param

I've got a case where I want to proxy a particular call down to a different backend based on the existence of a query param. The following is sort of what I start with
location ~ ^/abc/xyz/?$ {
proxy_pass $backend_url;
}
What I'd like to do is check for a query param foo (or even just the existence of that string anywhere). So I thought I could do this
location ~ ^/abc/xyz/?$ {
set $backend_url "somelocation"
if ($request_url ~ .*foo.*) {
set $backend_url "someotherlocation"
proxy_pass $backend_url
}
proxy_pass $backend_url;
}
But this doesn't seem to actually proxy to the new location. Am I doing something wrong with my code, or is the whole approach wrong?
I don't know why are you using two proxy_pass directives, this block should do it in a logic you described:
location ~ ^/abc/xyz/?$ {
set $backend_url "somelocation";
if ($request_url ~ \?(.*&)?foo(=|&|$)) {
set $backend_url "otherlocation";
}
proxy_pass $backend_url;
}
I slightly modified your regex to match only request URLs where foo is a query argument name and not a query argument value (or its part). However I'd rather use map directive for this purpose (map block should be placed outside the server block):
map $arg_foo $backend_url {
"" somelocation; # if the value is empty
default otherlocation; # otherwise
}
server {
...
location ~ ^/abc/xyz/?$ {
proxy_pass $backend_url;
}
}
Pay an attention you may need to define a resolver for this configuration to work (some additional info about this can be found here).

How to forward the URL's all parameters through a proxy_pass with nginx?

How to forward the URL's all parameters through a proxy_pass with nginx?
Nginx config:
location /proxy/ {
if ($request_method = HEAD) { return 200; }
if ( $arg_address != "" ) {
proxy_pass $arg_address;
return 301 $arg_address;
}
proxy_ssl_verify off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
this urls works:
https://localhost/proxy/?address=https://exemple.com/transfer/file.txt ==>> https://exemple.com/transfer/file.txt
or
https://localhost/proxy/?address=https://exemple.com/transfer/file.txt?host-id=1 ==>> https://exemple.com/transfer/file.tx?host-id=1
if I add multiple parameters, it will be truncated to the first "&"
https://localhost/proxy/?address=https://exemple.com/transfer/file.txt?host-id=1&password=123456&date=xxxxxx
==>> https://exemple.com/transfer/file.txt?host-id=1
How can I transfer the entire url?
Before we get to the answer, may I ask what do you want to achieve? Do you want to proxy the request or to generate HTTP 301 redirect? With the following construction
if ( $arg_address != "" ) {
proxy_pass $arg_address;
return 301 $arg_address;
}
you'll always get a redirect because the directives from ngx_http_rewrite_module are executed before any others, so the proxy_pass directive is useless here. ngx_http_rewrite_module is very special and different from most of the other modules. Although nginx configuration in general is declarative, rewrite module evaluates its instructions imperatively. This is always a source of confusion for every nginx novice. You can read more about the rewrite module internal implementation here.
If you want to proxy the request instead of generating a redirect, you'll need to remove return and add a resolver directive to your configuration. Here you can read why it is required.
A "dirty hack" solution
Being that said, get back to the question. Of course, when nginx receives the request
https://localhost/proxy/?address=https://example.com/transfer/file.txt?host-id=1&password=123456&date=20210520
arg_NAME variables will be filled the following way:
arg_address => https://example.com/transfer/file.txt?host-id=1
arg_password => 123456
arg_date => 20210520
It is correct and expected behavior.
What you can do to preserve all the other query arguments? The most simple is to assume that all query arguments following the address one are subject to pass to the upstream and use a map directive to get the required string:
map $args $address {
~(?:^|&)(address=.*) $1;
}
server {
...
location /proxy/ {
...
if ($address) {
# 'proxy_pass $address' or 'return 301 $address' here
}
...
}
...
}
Here is more strict check where we take the rest of the query string only if there is a question mark after address query parameter and only $arg_address value otherwise:
map $args $address {
~(?:^|&)(address=[^&?]+\?.*) $1;
default $arg_address;
}
Reliable solution
While the answer above is generally workable, I'd rather try to design my proxy solution using URL encoding on address query argument to avoid reserved characters usage as part of the query argument value. The above request being URL-encoded would look like
https://localhost/proxy/?address=https%3A%2F%2Fexample.com%2Ftransfer%2Ffile.txt%3Fhost-id%3D1%26password%3D123456%26date%3D20210520
The bad thing is that "vanilla" nginx doesn't have an ability to URL-decode an arbitrary string. However it can be done using OpenResty/lua-nginx-module:
location /proxy/ {
...
set_by_lua_block $address { return ngx.unescape_uri(ngx.var.arg_address) }
if ($address) {
# 'proxy_pass $address' or 'return 301 $address' here
}
...
}
or set-misc-nginx-module:
location /proxy/ {
...
if ($arg_address) {
set_unescape_uri $address $arg_address;
# 'proxy_pass $address' or 'return 301 $address' here
}
...
}
Perhaps the same can be done using njs, but I didn't use it and can't give you an example.

How to get nginx to do a redirect to url-encoded query parameter

I have a requirement to do a proxy call to url delivered via a query parameter as per example:
My nginx proxy is deployed at: https://myproxy.net
if the redirect parameter is not url encoded I can do the call with this block:
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass $arg_redirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
the error intercepts and #handle_redirects then take care of othe 30X codes that might pop up at new destination.
This works for a request:
GET: https://myproxy.net/basepath?redirect=https://destination.com/somepath/uuid
What do I need to do to make it work for:
GET: https://myproxy.net/basepath?redirect=https%3A%2F%2Fdestination.com%2Fsomepath%2Fuuid
Additionally as part of spec it has to be pure nginx, not additional modules, lua etc.
Thanks!
Actually, proxy_pass does normalisation by default, but it only affects $uri part. Thus you only need to decode the beginning of the passed string to get it working:
location / {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
if ( $arg_redirect ~ (.+)%3A%2F%2F(.+) ){ # fix :// between scheme and destination
set $arg_redirect $1://$2;
}
if ( $arg_redirect ~ (.+?)%3A(.*) ){ # fix : between destination and port
set $arg_redirect $1:$2;
}
if ( $arg_redirect ~ (.+?)%2F(.*) ){ # fix / after port, the rest will be decoded by proxy_pass
set $arg_redirect $1/$2;
}
proxy_pass $arg_redirect;
}
With the above I managed to access http://localhost/?redirect=http%3A%2F%2F127.0.0.1%3A81%2Fsfoo%20something%2Fs
The solution seems dirty and the only alternative using default modules is map (even less cleaner in my opinion). I'd rather split redirect argument into pieces: scheme (http or https), destination, port, and uri. With that you would be able to construct full address without rewriting:
proxy_pass $arg_scheme://$arg_dest:$arg_port/$arg_uri
Ok, there is very weird and curious solution
server {
listen 80;
resolver x.x.x.x;
location /basepath {
if ($arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass http://127.0.0.1:80/basepath/$arg_redirect;
}
location ~ ^/basepath/(?<proto>\w+):/(?<redir>.+)$ {
proxy_pass $proto://$redir;
}
}
Nginx does not encode path with variables in proxy_pass and send it as is. So, I make $arg_* part of proxy_pass uri, send request to self and nginx will receive new request which will be decoded.
But because Nginx will clean path and replace // to / I split protocol part in regexp.
And ... I would never recommend using this solution, but it works :)
try like this and let me know if it works
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
set_unescape_uri $decodedredirect $arg_redirect;
proxy_pass $decodedredirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}

nginx as rate limiter based on http body

I'm evaluating nginx to act as rate limiter for a multi tenancy REST API system. I need to limit API calls by tenant-id.
For example i want to allow 100 r/s for tenant1 and only 50 r/s for tenant2.
It can be easily achived when there are differant urls like: "me.com/tenant1/api" and "me.com/tenant2/api" (with the location directive).
But, in my case the urls are the same for all tenants "me.com/api" (I can't change this).
To find the tenant-id I need to extract a JSON attribute from the Body of the request, and then check the DB for the real tenant-id.
Is it possible to limit_req with my requirements?
Thank for the help!
I decided to build another service getTenant for parsing the body and extracting the Tenant from the DB. This service is called internally by Nginx.
I'm not sure if that is the best nginx (/openresty) solution, but this is what i came up with:
limit_req_zone t1Limit zone=t1Zone:10m rate=200r/s;
limit_req_zone t2Limit zone=t2Zone:10m rate=90r/s;
server {
location /api{
content_by_lua_block {
ngx.req.read_body();
local reqBody = ngx.req.get_body_data()
local res = ngx.location.capture("/getTenant", {method=ngx.HTTP_POST,body=reqBody});
local tenantId= res.body;
if tenantId== "none" then
ngx.log(ngx.ERR, "Tenant not found!");
ngx.say(tenantId);
else
ngx.req.set_header("x_myTenantId", tenantId)
local res2 = ngx.location.capture("/" .. tenantId .."/doApi", {method=ngx.HTTP_POST,body=reqBody});
if res2.status == ngx.HTTP_OK then
ngx.say(res2.body);
ngx.exit(res2.status);
else
ngx.status = res2.status
ngx.exit(res2.status)
end
end;
}
}
location /getTenant {
internal; #this is not accessible from outside.
proxy_pass http://UpStream1/getCustomer;
proxy_set_header X-Original-URI $request_uri;
}
location /tenant1/doApi {
internal; #this is not accessible from outside.
# Proxy all requests to the AReqUpStream server group
proxy_pass http://UpStream2/doApi;
limit_req zone=tenant1Zone burst=25;
limit_req_log_level notice;
}
location /tenant2/doApi {
internal; #this is not accessible from outside.
# Proxy all requests to the AReqUpStream server group
proxy_pass http://UpStream2/doApi;
limit_req zone=tenant2Zone burst=10 ;#nodelay;
limit_req_status 409;
limit_req_log_level notice;
}
}
Basically, when me.com/api is called, a new subrequest is issued to service /getTenant. The response of that call is used to build another subrequest call to the /tenant[X]/doApi service. That way i can define locations per tenant and provide different rate_limis to each.
Comments on that are more than welcome!

How should I configure Nginx to proxy to a URL passed by parameter?

I'm trying to get access to media files (images, videos) sitting behind an OAuth2 authentication.
In order to access the resource I need to add a custom Authorization Bearer token to the request, so I can't use a simple rewrite (well, as far as I know at least).
It cannot be done via plain HTML (say img or video tag) so I'm considering to have Nginx proxying the queries to the final server.
Each of the media resources would be loaded via a /proxy path, with a token parameter (for authentication) and url for the actual resource to load.
Sample URL:
http://myserver.com/proxy/?token=12345&url=http://protectedserver.com/custompath/asset
This is what I came up with but I am not quite sure how to configure the proxy_pass directive since I need it to proxy to the $url variable specifically. I do not need to proxy the path (which would be empty anyway).
location /proxy/ {
if ($arg_token ~ "^$") { return 404; }
if ($arg_url ~ "^$") { return 404; }
set $url $arg_url;
proxy_set_header Authorization "Bearer $arg_token";
set $args "";
#proxy_pass $url;
}
Note: this will be run in a closed environment and only specific machines (kiosks with limited interaction) will be able to access the page so I'm not concerned about a potential leak of the auth token.
I noticed a similar question on ServerFault, but no one had an answer to that:
https://serverfault.com/questions/671991/nginx-proxy-pass-url-from-get-argument
I'm looking for a config setting to make it work or a viable alternative solution.
Here is a correct configuration for my problem:
location /proxy/ {
if ($arg_token ~ "^$") { return 404; }
if ($arg_url ~ "^$") { return 404; }
set $url $arg_url;
set $token $arg_token;
set $args "";
# IMPORTANT, this is required when using dynamic proxy pass
# You can alternatively use any DNS resolver under your control
resolver 8.8.8.8;
proxy_pass $url;
proxy_set_header Authorization "Bearer $token";
proxy_redirect off;
}

Resources