Fetch information from an API before sending the request upstream - nginx

Is it possible to send a http subrequest in a location block and use the response in the proxy_pass directive?
use case
My upstream application needs some additional information from an API.
I've written a location block that proxies request with the proxy_pass directive.
Before nginx sends the request to my application. I'd like to send an HTTP request to my API and use several response headers
as request headers to my application.
This is the outline of what I want to achieve:
server {
server_name ...;
location {
# perform subrequest to fetch additional information from an api
proxy_pass myapplication;
proxy_set_header X-Additional-Info "some information from the subrequest";
}
}
The behaviour is similar to the auth_request module. However, I can't find documentation of sending an additional blocking HTTP request before inside a location block using standard nginx configuration.

You can't do it using regular nginx directives but it's quite easy using lua-nginx-module.
This module embeds Lua, via the standard Lua 5.1 interpreter or LuaJIT
2.0/2.1, into Nginx and by leveraging Nginx's subrequests, allows the integration of the powerful Lua threads (Lua coroutines) into the
Nginx event model.
Here's how to accomplish what you need:
create a directory conf.d/
put 2 files test.conf and header.lua into it (see the contents below)
docker run -p8080:8080 -v your_path/conf.d:/etc/nginx/conf.d openresty/openresty:alpine
curl http://localhost:8080/
test.conf
server {
listen 8080;
location /fetch_api {
# this is a service echoing your IP address
proxy_pass http://api.ipify.org/;
}
location / {
set $api_result "";
access_by_lua_file /etc/nginx/conf.d/header.lua;
proxy_set_header X-Additional-Info $api_result;
# this service just prints out your request headers
proxy_pass http://scooterlabs.com/echo;
}
}
header.lua
local res = ngx.location.capture('/fetch_api', { method = ngx.HTTP_GET, args = {} });
ngx.log(ngx.ERR, res.status);
if res.status == ngx.HTTP_OK then
ngx.var.api_result = res.body;
else
ngx.exit(403);
end
results
curl http://localhost:8080/
Simple webservice echo test: make a request to this endpoint to return the HTTP request parameters and headers. Results available in plain text, JSON, or XML formats. See http://www.cantoni.org/2012/01/08/simple-webservice-echo-test for more details, or https://github.com/bcantoni/echotest for source code.
Array
(
[method] => GET
[headers] => Array
(
[X-Additional-Info] => my-ip-address
[Host] => scooterlabs.com
[Connection] => close
[User-Agent] => curl/7.43.0
[Accept] => */*
)
[request] => Array
(
)
[client_ip] => my-ip-address
[time_utc] => 2018-01-23T19:25:56+0000
[info] => Echo service from Scooterlabs (http://www.scooterlabs.com)
)
Notice the X-Additional-Info header populated with the data obtained in /fetch_api handler

Related

Nginx authenticate against backend server with j_security_check

I need a reverse proxy for a backend Rest API to avoid CORS and debugging issues while developing a Svelte App. I've tried a few options like Node JS Express with http-proxy-middleware, Nginx, etc. The problem is that the backend JAX-RS Rest API requires j_security_check authentication. My modest network protocol skills run out trying to configure that in the proxy...
How would I make the following work:
Svelte App (http:5000) -> Proxy (http:3333) -> Rest API (http:8080 w/j_security_check)
So for example:
Svelte:
const ping = async () => {
const response = await fetch('http://localhost:3333/api/ping', {
method: 'GET',
headers: {
Accept: 'application/json',
},
});
return await response.json();
};
Nginx so far:
location /api/ {
add_header Access-Control-Allow-Origin "http://localhost:5000";
add_header Access-Control-Allow-Methods "GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH";
proxy_pass http://backend:8080/;
proxy_set_header Host $host;
}
I guess I'm missing a lot of header config from the above. If it's even possible for j_security_check.
Since this is local development proxy only, it would be optimal if the proxy could handle the authentication and leave those details hidden from the Svelte App.

NGINX read body from proxy_pass response

I have two servers:
NGINX (it exchanges file id to file path)
Golang (it accepts file id and return it's path)
Ex: When browser client makes request to https://example.com/file?id=123, NGINX should proxy this request to Golang server https://go.example.com/getpath?file_id=123, which will return the response to NGINX:
{
data: {
filePath: "/static/..."
},
status: "ok"
}
Then NGINX should get value from filePath and return file from the location.
So the question is how to read response (get filePath) in NGINX?
I assume you are software developer and your have full control over your application so there is no need to force square peg in a round hole here.
Different kinds of reverse proxies support ESI(Edge Side Includes) technology which allow developer to replace different parts of responce body with content of static files or with response bodies from upstream servers.
Nginx has such technology as well. It is called SSI (Server Side Includes).
location /file {
ssi on;
proxy_pass http://go.example.com;
}
Your upstream server can produce body with content <!--# include file="/path-to-static-files/some-static-file.ext" --> and nginx will replace this in-body directive with content of the file.
But you mentioned streaming...
It means that files will be of arbitrary sizes and building response with SSI would certainly eat precious RAM resources so we need a Plan #B.
There is "good enough" method to feed big files to the clients without showing static location of the file to the client.
You can use nginx's error handler to server static files based on information supplied by upstream server.
Upstream server for example can send back redirect 302 with Location header field containing real file path to the file.
This response does not reach the client and is feed into error handler.
Here is an example of config:
location /file {
error_page 302 = #service_static_file;
proxy_intercept_errors on;
proxy_set_header Host $host;
proxy_pass http://go.example.com;
}
location #service_static_file {
root /hidden-files;
try_files $upstream_http_location 404.html;
}
With this method you will be able to serve files without over-loading your system while having control over whom do you give the file.
For this to work your upstream server should respond with status 302 and with typical "Location:" field and nginx will use location content to find the file in the "new" root for static files.
The reason for this method to be of "good enough" type (instead of perfect) because it does not support partial requests (i.e. Range: bytes ...)
Looks like you are wanting to make an api call for data to run decision and logic against. That's not quite what proxying is about.
The core proxy ability of nginx is not designed for what you are looking to do.
Possible workaround: extending nginx...
Nginx + PHP
Your php code would do the leg work.
Serve as a client to connect to the Golang server and apply additional logic to the response.
<?php
$response = file_get_contents('https://go.example.com/getpath?file_id='.$_GET["id"]);
preg_match_all("/filePath: \"(.*?)\"/", $response, $filePath);
readfile($filePath[1][0]);
?>
location /getpath {
try_files /getpath.php;
}
This is just the pseudo-code example to get it rolling.
Some miscellaneous observations / comments:
The Golang response doesn't look like valid json, replace preg_match_all with json_decode if so.
readfile is not super efficient. Consider being creative with a 302 response.
Nginx + Lua
sites-enabled:
lua_package_path "/etc/nginx/conf.d/lib/?.lua;;";
server {
listen 80 default_server;
listen [::]:80 default_server;
location /getfile {
root /var/www/html;
resolver 8.8.8.8;
set $filepath "/index.html";
access_by_lua_file /etc/nginx/conf.d/getfile.lua;
try_files $filepath =404;
}
}
Test if lua is behaving as expected:
getfile.lua (v1)
ngx.var.filepath = "/static/...";
Simplify the Golang response body to just return a bland path then use it to set filepath:
getfile.lua (v2)
local http = require "resty.http"
local httpc = http.new()
local query_string = ngx.req.get_uri_args()
local res, err = httpc:request_uri('https://go.example.com/getpath?file_id=' .. query_string["id"], {
method = "GET",
keepalive_timeout = 60,
keepalive_pool = 10
})
if res and res.status == ngx.HTTP_OK then
body = string.gsub(res.body, '[\r\n%z]', '')
ngx.var.filepath = body;
ngx.log(ngx.ERR, "[" .. body .. "]");
else
ngx.log(ngx.ERR, "missing response");
ngx.exit(504);
end
resty.http
mkdir -p /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http_headers.lua" -P /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http.lua" -P /etc/nginx/conf.d/lib/resty

Nginx - Decode URL query parameter and forward it as request header

I need to send some basic auth credentials (es. user:pass) to nginx in the form of query parameter (es. http://example.com?BASIC_AUTH=dXNlcjpwYXNz) and being able to forward them in the more usual Authorization: Basic dXNlcjpwYXNz header form to a target server behind the proxy.
I'm already able to retrieve the value of the encoded auth string with a regular expression. The problem is that very often that value may contain some character that need to be percent-encoded in the URL. Es. user:pass! -> ?BASIC_AUTH=dXNlcjpwYXNzIQ== becomes ?BASIC_AUTH=dXNlcjpwYXNzIQ%3D%3D
Therefore, when I forward the request to the target server, I end up specifing Authorization: Basic dXNlcjpwYXNzIQ%3D%3D which the target server will reject, giving a 401 Unauthorized.
How can I force nginx to decode the auth string before setting the Authorization header? Thanks in advance for your help.
Note: I can't send the auth string in the Authorization header in the first place due to some application-specific constraints.
"Pure" nginx solution
Unfortunately nginx does not provide a rich string operations set. I think there isn't a way to do global search-and-replace through some string (which can be a solution if we could replace all %2B with +, %2F with / and %3D with =). However there are circumstances under which nginx performs an urldecoding of some string - when this string becomes a part of an URI which will be forwarded to an upstream proxy server.
So we can add a value of a BASIC_AUTH request argument to the URI and make a proxy request to ourself:
# Main server block
server {
listen 80 default_server;
...
location / {
if ($arg_basic_auth) {
# "basic_auth" request argument is present,
# append "/decode_basic_auth/<BASE64_token>" to the URI
# and go to the next location block
rewrite ^(.*)$ /decode_basic_auth/$arg_basic_auth$1 last;
}
# No "basic_auth" request argument present,
# can do a proxy call from here without setting authorization headers
...
}
location /decode_basic_auth/ {
# This will be an internal location only
internal;
# Remove "basic_auth" request argument from the list of arguments
if ($args ~* (.*)(^|&)basic_auth=[^&]*(\2|$)&?(.*)) {
set $args $1$3$4;
}
# Some hostname for processing proxy subrequests
proxy_set_header Host internal.basic.auth.localhost;
# Do a subrequest to ourselfs, preserving other request arguments
proxy_pass http://127.0.0.1$uri$is_args$args;
}
}
# Additional server block for proxy subrequests processing
server {
listen 80;
server_name internal.basic.auth.localhost;
# Got URI in form "/decode_basic_auth/<BASE64_token>/<Original_URI>"
location ~ ^/decode_basic_auth/([^/]+)(/.*)$ {
proxy_set_header Authorization "Basic $1";
# Setup other HTTP headers here
...
proxy_pass http://<upstream_server>$2$is_args$args;
}
# Do not serve other requests
location / {
return 444;
}
}
Maybe this is not a very elegant solution, but it is tested and works.
OpenResty / ngx_http_lua_module
This can be easily solved with openresty or ngx_http_lua_module using ngx.escape_uri function:
server {
listen 80 default_server;
...
location / {
set $auth $arg_basic_auth;
if ($args ~* (.*)(^|&)basic_auth=[^&]*(\2|$)&?(.*)) {
set $args $1$3$4;
}
rewrite_by_lua_block {
ngx.var.auth = ngx.unescape_uri(ngx.var.auth)
}
proxy_set_header Authorization "Basic $auth";
# Setup other HTTP headers here
...
proxy_pass http://<upstream_server>;
}
}

Nginx - How can I create a custom request that will be used with the auth_request module

To start with: I am NOT an nginx expert. Very much a newbie to it.
I am attempting to protect a 3rd party piece of software with nginx doing the authentication (really - just verifying that the request has a valid OAuth2 Bearer token)
The HTTP request will have an OAuth2 bearer token in the Authentication header.
e.g. Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZ....H5w
I have an OAuth2 server (UAA) that has an api where I can call http://myuaa/check_token?token=eyJhbGciOiJSUzI1NiIsImtpZ....H5w to get back a 2XX or a 4XX if the token is valid. A complication is that this server does require basic auth to call the /check_token endpoint.
I have tried using a map to parse the token from the authorization header, but with no luck.
Just kind of at a loss.
Perhaps this isn't a good fit for Nginx?
relevant pieces of the nginx.conf
# this map isnt working as I thought it might
http {
...
map $http_authorization $token {
~Bearer(?<token>abc) $token;
}
...
# test just to see if the authorization header is being parsed and passed - no luck
location /oauth {
proxy_set_header X-my-header $token;
proxy_set_header X-another-header value;
proxy_set_header Authorization "Basic basdasdfasdf";
proxy_pass http://localhost:8080;
}
Expected request to the 3rd party server that nginx is protecting:
<GET|POST|PUT|DELETE> /anyurl HTTP1/1.1
..
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZ....H5w
..
Expected request forwarded to the UAA server to validate token
GET /check_token?token=eyJhbGciOiJSUzI1NiIsImtpZ....H5w
..
Authorization Basic asfasdfdf
..
Your map directive isn't working, named group token somehow interfere with the $token variable, any of these definitions would work:
map $http_authorization $token {
~^Bearer\s+([\S]+)$ $1;
}
or
map $http_authorization $token {
~^Bearer\s+(?<bearer>[\S]+)$ $bearer;
}
Full working config will be looking like this:
map $http_authorization $token {
~^Bearer\s+(?<bearer>[\S]+)$ $bearer;
}
server {
...
location / {
auth_request /uaa;
...
}
location /uaa {
internal;
proxy_pass_request_body off;
proxy_set_header Authorization "Basic your_base64_auth_string";
proxy_set_header Content-Length "";
proxy_pass http://localhost:8080/check_token?token=$token;
}
}

How to change request parameters before passing request to nginx reverse proxy server

I am using Nginx to proxy request to some backend server.
For this, i am using proxy_pass module to redirect.
Config file:
location = /hello {
proxy_pass: abc.com
}
I want to execute the following workflow.
Request to nginx server --> change request parameters --> pass the request to abc.com --> change the response parameters --> send response back to client.
Is this possible with nginx ? Any help/pointers to this problem would be appreciated.
You should be able to change/set new parameters with this
location /hello {
proxy_pass: abc.com
if ($args ~* paramToChange=(.+)) {
set $args newParamName=$1;
}
set $args otherParam=value;
}
Update:
There is not a way in nginx out of the box to make a request to get params dynamically then apply them to another request before sending the client response.
You can do this by adding a module to nginx called lua.
This module can be recompiled into nginx by downloading it and adding it in the .configure options during installation. Also I like the openresty bundle that comes with it and other useful modules already available, like echo.
Once you have the lua module this server code will work:
server {
listen 8083;
location /proxy/one {
proxy_set_header testheader test1;
proxy_pass http://localhost:8081;
}
location /proxy/two {
proxy_pass http://localhost:8082;
}
location / {
default_type text/html;
content_by_lua '
local first = ngx.location.capture("/proxy/one",
{ args = { test = "test" } }
)
local testArg = first.body
local second = ngx.location.capture("/proxy/two",
{ args = { test = testArg } }
)
ngx.print(second.body)
';
}
}
I tested this configuration with a couple of node js servers like this:
var koa = require('koa');
var http = require('http');
startServerOne();
startServerTwo();
function startServerOne() {
var app = koa();
app.use(function *(next){
console.log('\n------ Server One ------');
console.log('this.request.headers.testheader: ' + JSON.stringify(this.request.headers.testheader));
console.log('this.request.query: ' + JSON.stringify(this.request.query));
if (this.request.query.test == 'test') {
this.body = 'First query worked!';
}else{
this.body = 'this.request.query: ' + JSON.stringify(this.request.query);
}
});
http.createServer(app.callback()).listen(8081);
console.log('Server 1 - 8081');
}
function startServerTwo(){
var app = koa();
app.use(function *(next){
console.log('\n------ Server Two ------');
console.log('this.request.query: ' + JSON.stringify(this.request.query));
if (this.request.query.test == 'First query worked!') {
this.body = 'It Worked!';
}else{
this.body = 'this.request.query: ' + JSON.stringify(this.request.query);
}
});
http.createServer(app.callback()).listen(8082);
console.log('Server 2 - 8082');
}
This was the output from the node console logs:
Server 1 - 8081
Server 2 - 8082
------ Server One ------
this.request.headers.testheader: "test1"
this.request.query: {"test":"test"}
------ Server Two ------
this.request.query: {"test":"First query worked!"}
Here's what happens:
Nginx sends server one a request query with the test parameter set.
Node server 1 sees the test parameter and responds with 'First query worked!'.
Nginx updates the query parameters with the body from the server one response.
Nginx sends server two a request with the new query parameters.
Node server 2 sees that the 'test' query parameter equals 'First query worked!' and responds to the request with response body 'It Worked!'.
And the curl response or visiting localhost:8083 in a browser shows 'It worked':
curl -i 'http://localhost:8083'
HTTP/1.1 200 OK
Server: openresty/1.9.3.2
Date: Thu, 17 Dec 2015 16:57:45 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
It Worked!

Resources