I need to send some basic auth credentials (es. user:pass) to nginx in the form of query parameter (es. http://example.com?BASIC_AUTH=dXNlcjpwYXNz) and being able to forward them in the more usual Authorization: Basic dXNlcjpwYXNz header form to a target server behind the proxy.
I'm already able to retrieve the value of the encoded auth string with a regular expression. The problem is that very often that value may contain some character that need to be percent-encoded in the URL. Es. user:pass! -> ?BASIC_AUTH=dXNlcjpwYXNzIQ== becomes ?BASIC_AUTH=dXNlcjpwYXNzIQ%3D%3D
Therefore, when I forward the request to the target server, I end up specifing Authorization: Basic dXNlcjpwYXNzIQ%3D%3D which the target server will reject, giving a 401 Unauthorized.
How can I force nginx to decode the auth string before setting the Authorization header? Thanks in advance for your help.
Note: I can't send the auth string in the Authorization header in the first place due to some application-specific constraints.
"Pure" nginx solution
Unfortunately nginx does not provide a rich string operations set. I think there isn't a way to do global search-and-replace through some string (which can be a solution if we could replace all %2B with +, %2F with / and %3D with =). However there are circumstances under which nginx performs an urldecoding of some string - when this string becomes a part of an URI which will be forwarded to an upstream proxy server.
So we can add a value of a BASIC_AUTH request argument to the URI and make a proxy request to ourself:
# Main server block
server {
listen 80 default_server;
...
location / {
if ($arg_basic_auth) {
# "basic_auth" request argument is present,
# append "/decode_basic_auth/<BASE64_token>" to the URI
# and go to the next location block
rewrite ^(.*)$ /decode_basic_auth/$arg_basic_auth$1 last;
}
# No "basic_auth" request argument present,
# can do a proxy call from here without setting authorization headers
...
}
location /decode_basic_auth/ {
# This will be an internal location only
internal;
# Remove "basic_auth" request argument from the list of arguments
if ($args ~* (.*)(^|&)basic_auth=[^&]*(\2|$)&?(.*)) {
set $args $1$3$4;
}
# Some hostname for processing proxy subrequests
proxy_set_header Host internal.basic.auth.localhost;
# Do a subrequest to ourselfs, preserving other request arguments
proxy_pass http://127.0.0.1$uri$is_args$args;
}
}
# Additional server block for proxy subrequests processing
server {
listen 80;
server_name internal.basic.auth.localhost;
# Got URI in form "/decode_basic_auth/<BASE64_token>/<Original_URI>"
location ~ ^/decode_basic_auth/([^/]+)(/.*)$ {
proxy_set_header Authorization "Basic $1";
# Setup other HTTP headers here
...
proxy_pass http://<upstream_server>$2$is_args$args;
}
# Do not serve other requests
location / {
return 444;
}
}
Maybe this is not a very elegant solution, but it is tested and works.
OpenResty / ngx_http_lua_module
This can be easily solved with openresty or ngx_http_lua_module using ngx.escape_uri function:
server {
listen 80 default_server;
...
location / {
set $auth $arg_basic_auth;
if ($args ~* (.*)(^|&)basic_auth=[^&]*(\2|$)&?(.*)) {
set $args $1$3$4;
}
rewrite_by_lua_block {
ngx.var.auth = ngx.unescape_uri(ngx.var.auth)
}
proxy_set_header Authorization "Basic $auth";
# Setup other HTTP headers here
...
proxy_pass http://<upstream_server>;
}
}
Related
Example request - http://localhost/iframe?ip=192.168.0.237
I want to proxy pass the request to the value of IP and remove the path and args after localhost/ .
Ideally the proxy_pass should point to 192.168.0.237 and the URL should be http://localhost/.
localhost /iframe {
rewrite ^/(iframe/.*)$ http://localhost/ permanent;
proxy_pass $arg_ip;
}
I'm not sure whether rewrite is the proper way to address this problem.
I would use the argument ip and a rewrite to remove the iframe location
server {
listen 8085;
location /iframe {
rewrite ^/iframe(.*)$ /$1 break;
proxy_pass http://$arg_ip;
}
}
server {
listen 8080;
location / { return 200 "$host$uri"; }
}
Security Notice
I just have a feeling you should whilelist the upstream servers accepted as arguments. If not this will be a wildcard proxy to every single http-server reachable in the network. This is a easy to use SSRF attack vector. So please add some extra layer of security.
SSRF Explained:
Let's say we use this configuration without any further security. Given the folowing NGINX config:
server {
listen 8085;
location /iframe {
rewrite ^/iframe(.*)$ /$1 break;
proxy_pass http://$arg_ip;
}
}
# Server for iframe service
server {
listen 8080;
root /usr/share/nginx/;
location / { return 200 "$host$uri\n"; }
}
# Private Server Section here!
server {
listen 8086;
allow 127.0.0.1;
deny all;
.....
location / {
index welcome.html;
}
}
Trying to reach the secret server directly
curl -v EXTERNALIP:8086
will fail with HTTP 403.
The NGINX will just allow connections form localhost/127.0.0.1 as defined in the allow/deny directives.
But lets try the iframe with the ip argument.
$# curl localhost:8085/iframe?ip=127.0.0.1:8086
Welcome to our very secure server! Internals only!
It prints the content of the secret server. Whitlisting a proxy-pass like this is never a good idea regardless its working or not.
I have a requirement to do a proxy call to url delivered via a query parameter as per example:
My nginx proxy is deployed at: https://myproxy.net
if the redirect parameter is not url encoded I can do the call with this block:
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass $arg_redirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
the error intercepts and #handle_redirects then take care of othe 30X codes that might pop up at new destination.
This works for a request:
GET: https://myproxy.net/basepath?redirect=https://destination.com/somepath/uuid
What do I need to do to make it work for:
GET: https://myproxy.net/basepath?redirect=https%3A%2F%2Fdestination.com%2Fsomepath%2Fuuid
Additionally as part of spec it has to be pure nginx, not additional modules, lua etc.
Thanks!
Actually, proxy_pass does normalisation by default, but it only affects $uri part. Thus you only need to decode the beginning of the passed string to get it working:
location / {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
if ( $arg_redirect ~ (.+)%3A%2F%2F(.+) ){ # fix :// between scheme and destination
set $arg_redirect $1://$2;
}
if ( $arg_redirect ~ (.+?)%3A(.*) ){ # fix : between destination and port
set $arg_redirect $1:$2;
}
if ( $arg_redirect ~ (.+?)%2F(.*) ){ # fix / after port, the rest will be decoded by proxy_pass
set $arg_redirect $1/$2;
}
proxy_pass $arg_redirect;
}
With the above I managed to access http://localhost/?redirect=http%3A%2F%2F127.0.0.1%3A81%2Fsfoo%20something%2Fs
The solution seems dirty and the only alternative using default modules is map (even less cleaner in my opinion). I'd rather split redirect argument into pieces: scheme (http or https), destination, port, and uri. With that you would be able to construct full address without rewriting:
proxy_pass $arg_scheme://$arg_dest:$arg_port/$arg_uri
Ok, there is very weird and curious solution
server {
listen 80;
resolver x.x.x.x;
location /basepath {
if ($arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass http://127.0.0.1:80/basepath/$arg_redirect;
}
location ~ ^/basepath/(?<proto>\w+):/(?<redir>.+)$ {
proxy_pass $proto://$redir;
}
}
Nginx does not encode path with variables in proxy_pass and send it as is. So, I make $arg_* part of proxy_pass uri, send request to self and nginx will receive new request which will be decoded.
But because Nginx will clean path and replace // to / I split protocol part in regexp.
And ... I would never recommend using this solution, but it works :)
try like this and let me know if it works
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
set_unescape_uri $decodedredirect $arg_redirect;
proxy_pass $decodedredirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
I have two servers:
NGINX (it exchanges file id to file path)
Golang (it accepts file id and return it's path)
Ex: When browser client makes request to https://example.com/file?id=123, NGINX should proxy this request to Golang server https://go.example.com/getpath?file_id=123, which will return the response to NGINX:
{
data: {
filePath: "/static/..."
},
status: "ok"
}
Then NGINX should get value from filePath and return file from the location.
So the question is how to read response (get filePath) in NGINX?
I assume you are software developer and your have full control over your application so there is no need to force square peg in a round hole here.
Different kinds of reverse proxies support ESI(Edge Side Includes) technology which allow developer to replace different parts of responce body with content of static files or with response bodies from upstream servers.
Nginx has such technology as well. It is called SSI (Server Side Includes).
location /file {
ssi on;
proxy_pass http://go.example.com;
}
Your upstream server can produce body with content <!--# include file="/path-to-static-files/some-static-file.ext" --> and nginx will replace this in-body directive with content of the file.
But you mentioned streaming...
It means that files will be of arbitrary sizes and building response with SSI would certainly eat precious RAM resources so we need a Plan #B.
There is "good enough" method to feed big files to the clients without showing static location of the file to the client.
You can use nginx's error handler to server static files based on information supplied by upstream server.
Upstream server for example can send back redirect 302 with Location header field containing real file path to the file.
This response does not reach the client and is feed into error handler.
Here is an example of config:
location /file {
error_page 302 = #service_static_file;
proxy_intercept_errors on;
proxy_set_header Host $host;
proxy_pass http://go.example.com;
}
location #service_static_file {
root /hidden-files;
try_files $upstream_http_location 404.html;
}
With this method you will be able to serve files without over-loading your system while having control over whom do you give the file.
For this to work your upstream server should respond with status 302 and with typical "Location:" field and nginx will use location content to find the file in the "new" root for static files.
The reason for this method to be of "good enough" type (instead of perfect) because it does not support partial requests (i.e. Range: bytes ...)
Looks like you are wanting to make an api call for data to run decision and logic against. That's not quite what proxying is about.
The core proxy ability of nginx is not designed for what you are looking to do.
Possible workaround: extending nginx...
Nginx + PHP
Your php code would do the leg work.
Serve as a client to connect to the Golang server and apply additional logic to the response.
<?php
$response = file_get_contents('https://go.example.com/getpath?file_id='.$_GET["id"]);
preg_match_all("/filePath: \"(.*?)\"/", $response, $filePath);
readfile($filePath[1][0]);
?>
location /getpath {
try_files /getpath.php;
}
This is just the pseudo-code example to get it rolling.
Some miscellaneous observations / comments:
The Golang response doesn't look like valid json, replace preg_match_all with json_decode if so.
readfile is not super efficient. Consider being creative with a 302 response.
Nginx + Lua
sites-enabled:
lua_package_path "/etc/nginx/conf.d/lib/?.lua;;";
server {
listen 80 default_server;
listen [::]:80 default_server;
location /getfile {
root /var/www/html;
resolver 8.8.8.8;
set $filepath "/index.html";
access_by_lua_file /etc/nginx/conf.d/getfile.lua;
try_files $filepath =404;
}
}
Test if lua is behaving as expected:
getfile.lua (v1)
ngx.var.filepath = "/static/...";
Simplify the Golang response body to just return a bland path then use it to set filepath:
getfile.lua (v2)
local http = require "resty.http"
local httpc = http.new()
local query_string = ngx.req.get_uri_args()
local res, err = httpc:request_uri('https://go.example.com/getpath?file_id=' .. query_string["id"], {
method = "GET",
keepalive_timeout = 60,
keepalive_pool = 10
})
if res and res.status == ngx.HTTP_OK then
body = string.gsub(res.body, '[\r\n%z]', '')
ngx.var.filepath = body;
ngx.log(ngx.ERR, "[" .. body .. "]");
else
ngx.log(ngx.ERR, "missing response");
ngx.exit(504);
end
resty.http
mkdir -p /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http_headers.lua" -P /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http.lua" -P /etc/nginx/conf.d/lib/resty
I have the local server running on 3000 port and it sends some POST request to nginx server. Nginx should check referer, if it is not coming from 127.0.0.1:3000 (with all subdomains) then return 403 Restricted,otherwise if it is valid redirect to 9200/errors/browser endpoint.
Currently, it is always redirecting regardless if the referer is valid or not. I know that in nginx if is evil , so if-else approach doesnt work here.
server {
listen 127.0.0.1:9999;
server_name localhost;
location / {
valid_referers none blocked server_names ~someaddress;
if ($invalid_referer) {
return 403;
}
# redirect to this endpoint if referer is valid
return 307 http://localhost:9200/errors/browser;
}
}
I should add
if ($invalid_referer != "1") {
return 403;
}
as it is the empty string according to the documentation
$invalid_referer
Empty string, if the “Referer” request header field value is considered valid, otherwise “1”.
I need to set proxy_pass for the below url pattern.
location ~ ^/hosts/bu/(.*)/app/(.*)$ {
proxy_pass http://appserver.cnma.com:3000/hosts/bu/$1/app/$2;
}
When I try with the URL it does not pass the second parameter correctly. Please let me know where I am going wrong.
Orig docs say:
If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI:
location /some/path/ {
proxy_pass http://127.0.0.1;
}
So there is enough:
proxy_pass http://appserver.cnma.com:3000$request_uri;
you can use named capture in nginx while using regular expressions
location ~ ^/hosts/bu/(.*)/app/(.*)$ {
proxy_pass http://appserver.cnma.com:3000/hosts/bu/$1/app/$2;
}
You can modify block to below like this
location ~ ^/hosts/bu/(?<loc1>.+)/app/(?<loc2>.+)$ {
proxy_pass http://appserver.cnma.com:3000/hosts/bu/$1/app/$loc2;
}