nginx proxy_pass and URL decoding - nginx

Original URL: /api/url%2Fencoded%2F/?with=queryParams
nginx:
location /api {
client_max_body_size 2G;
proxy_pass https://oursite;
}
With this configuration, I was able to preserve the URL encoding when passing through the proxy. If I add a "/" after "oursite", it will decode the URL.
Problem:
Now the URL after being proxied still contains "/api/". I need to remove "/api/" only while still preserving the URL encoded parts.

Not a long time ago there was identical question without an answer. In my opinion, you should rething api to not have such weird URLs. Another way is to have api on subdomain. – Alexey Ten Mar 11 '15 at 22:58
stackoverflow.com/q/28684300/1016033 – Alexey Ten Mar 11 '15 at 23:01
Year-old challenge accepted!
location /api/ {
rewrite ^ $request_uri;
rewrite ^/api/(.*) $1 break;
return 400;
proxy_pass http://127.0.0.1:82/$uri;
}
That's it, folks!
More details at Nginx pass_proxy subdirectory without url decoding, but it does work even with the query string, too:
% curl "localhost:81/api/url%2Fencoded%2F/?with=queryParams"
/url%2Fencoded%2F/?with=queryParams
%

Disclaimer: I am sure this looks like an hack - and maybe it is. It is using the auth-subrequest feature for something else than auth, but it works!
If you want to keep any url-encoded part after /api/ from the original $request_uri I use NJS to set a variable and use it afterwards in the proxy_pass
js_import /etc/nginx/conf.d/http.js; # Import your njs file here
js_set $encodedUrlPart 'empty'; # Define a variable
location ~* api\/(.*)$ {
auth_request /urlencode; #This will get executed before proxy_pass
proxy_pass http://127.0.0.1:82/$encodedUrlPart;
}
and the http.js can look like this
function urlencode(r){
let regex = "(?<=\/api\/)(.*$)";
let url = r.variables.request_uri # this holds the original, non touched url
let lastPart = url.match(regex);
r.variables.encodedUrlPart = lastPart;
r.log("The encoded url part: " + r.variables.encodedUrlPart);
r.return(200); // need to return 200 so the 'auth' doesn't fail
}
export default {urlencode};
Is this considered unsafe? We could do some checking in the njs part though!

Related

Regex returning wrong string/variable?

I'm trying to secure a HLS stream using the secure_link module in nginx. I'm doing this in two parts:
First the URL is a "SEO" friendly version, this gets rewritten to the secure_link URL which has the ?token=xxx&?expires=000.
This gets processed in the next block, and here I ONLY want to test a part of the URL (stream name in this case) and not the whole URI. I'm using a regex to get the part of the URL I need to test in a variable. But for some reason every time I execute a request I get the wrong variable here (as outputted in the debug return line) if I use the "SEO" friendly URL that is rewritten, I get the $1 from the original url, and not the $1 of the "new* (re-written) URL.
If I submit the manually rewritten request I get the correct output for that variable. So this is only happening when I'm using the first rewritten URL.
Can someone point me as to what I going on here and how I can fix this? I'm out of ideas.
Demo URL:
https://myserver.com/live/hls/PSBOBrr5bVsd3wPLMMQE7Q/1658978846/AdminTest_1080/index.m3u8
Nginx config:
location /live {
add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Expose-Headers' 'Content-Length' always;
rewrite /hls/([a-zA-Z0-9_\-]*)/([0-9]*)/([a-zA-Z0-9_\-]*)/(.*)\.(ts|key|m3u8)$ /hls/$3/$4.$5?token=$1&expires=$2;
root /mnt/not-exist;
}
location ~ "/hls/([a-zA-Z0-9_\-]*)/" {
# location ~ /hls/(?<alias>[a-zA-Z0-9_]+)/ { #TRIED BOTH REGEX HERE, BOTH SAME RESULT
#internal;
secure_link $arg_token,$arg_expires;
secure_link_md5 "$host $1 $secure_link_expires $remote_addr secret";
# if ($secure_link = "") { return 403; }
# if ($secure_link = "0") { return 410; }
add_header Content-Type text/plain;
return 200 "secure_link: $host $1 $secure_link_expires $remote_addr "; # FOR DEBUGGING
}
When submitting the above example to the first location block I get the following output:
secure_link: myserver.com PSBOBrr5bVsd3wPLMMQE7Q 127.0.0.1
Which is clearly incorrectly returning another part of the URL ($1 from the first location block, not the current location block)
When using a manually crafted URL:
https://myserver.com/hls/AdminTest_1080/index.m3u8?token=token&expires=0000
I get the following (correct output):
secure_link: myserver.com AdminTest_1080 127.0.0.1
At this point I'm even considering that this might be a bug? Hoping for valuable input from the community!

How to get nginx to do a redirect to url-encoded query parameter

I have a requirement to do a proxy call to url delivered via a query parameter as per example:
My nginx proxy is deployed at: https://myproxy.net
if the redirect parameter is not url encoded I can do the call with this block:
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass $arg_redirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
the error intercepts and #handle_redirects then take care of othe 30X codes that might pop up at new destination.
This works for a request:
GET: https://myproxy.net/basepath?redirect=https://destination.com/somepath/uuid
What do I need to do to make it work for:
GET: https://myproxy.net/basepath?redirect=https%3A%2F%2Fdestination.com%2Fsomepath%2Fuuid
Additionally as part of spec it has to be pure nginx, not additional modules, lua etc.
Thanks!
Actually, proxy_pass does normalisation by default, but it only affects $uri part. Thus you only need to decode the beginning of the passed string to get it working:
location / {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
if ( $arg_redirect ~ (.+)%3A%2F%2F(.+) ){ # fix :// between scheme and destination
set $arg_redirect $1://$2;
}
if ( $arg_redirect ~ (.+?)%3A(.*) ){ # fix : between destination and port
set $arg_redirect $1:$2;
}
if ( $arg_redirect ~ (.+?)%2F(.*) ){ # fix / after port, the rest will be decoded by proxy_pass
set $arg_redirect $1/$2;
}
proxy_pass $arg_redirect;
}
With the above I managed to access http://localhost/?redirect=http%3A%2F%2F127.0.0.1%3A81%2Fsfoo%20something%2Fs
The solution seems dirty and the only alternative using default modules is map (even less cleaner in my opinion). I'd rather split redirect argument into pieces: scheme (http or https), destination, port, and uri. With that you would be able to construct full address without rewriting:
proxy_pass $arg_scheme://$arg_dest:$arg_port/$arg_uri
Ok, there is very weird and curious solution
server {
listen 80;
resolver x.x.x.x;
location /basepath {
if ($arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass http://127.0.0.1:80/basepath/$arg_redirect;
}
location ~ ^/basepath/(?<proto>\w+):/(?<redir>.+)$ {
proxy_pass $proto://$redir;
}
}
Nginx does not encode path with variables in proxy_pass and send it as is. So, I make $arg_* part of proxy_pass uri, send request to self and nginx will receive new request which will be decoded.
But because Nginx will clean path and replace // to / I split protocol part in regexp.
And ... I would never recommend using this solution, but it works :)
try like this and let me know if it works
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
set_unescape_uri $decodedredirect $arg_redirect;
proxy_pass $decodedredirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}

Is possible redirect by microservice?

Can I set a variable with an URL returned by a proxy? ... I want to avoid to run Java, PHP, Python etc. Need somethong simple and faster.
Note, to answer comments: "... an URL returned by a proxy" = a microservice that is a black-box returning the URL. Any URL, can be aleatory or function of inputs (passed to the proxy).
If it is possible, how to?
This is fine on my NGINX server, it is returning a string with the necessary URL.
location /_test {
rewrite ^/_test/(.*) /$1 break;
proxy_pass http://127.0.0.1:3000;
}
(here $1 is the input and 127.0.0.1:3000 the black-box microservice)
... How to redirect as return 301 $theNewUrl? Imagining something (illustrative and wrong of course) as
location /_test {
rewrite ^/_test/(.*) /$1 break;
set theNewUrl = proxy_pass(http://127.0.0.1:3000/$1);
return 301 $theNewUrl;
}
To redirect base on path on Ningx, try this:
example.com is your base domain.
server {
server_name myDomain;
location /_test/hello/ {
proxy_pass http://google.com/;
}
location /_test/bye/ {
proxy_pass http://stackoverflow.com/;
}
}
Example:
HTTP request to http://myDomain/_test/hello/$1 will be translated to: http://google.com/$1 you can use it with ports or whatever you want.

Nginx - Decode URL query parameter and forward it as request header

I need to send some basic auth credentials (es. user:pass) to nginx in the form of query parameter (es. http://example.com?BASIC_AUTH=dXNlcjpwYXNz) and being able to forward them in the more usual Authorization: Basic dXNlcjpwYXNz header form to a target server behind the proxy.
I'm already able to retrieve the value of the encoded auth string with a regular expression. The problem is that very often that value may contain some character that need to be percent-encoded in the URL. Es. user:pass! -> ?BASIC_AUTH=dXNlcjpwYXNzIQ== becomes ?BASIC_AUTH=dXNlcjpwYXNzIQ%3D%3D
Therefore, when I forward the request to the target server, I end up specifing Authorization: Basic dXNlcjpwYXNzIQ%3D%3D which the target server will reject, giving a 401 Unauthorized.
How can I force nginx to decode the auth string before setting the Authorization header? Thanks in advance for your help.
Note: I can't send the auth string in the Authorization header in the first place due to some application-specific constraints.
"Pure" nginx solution
Unfortunately nginx does not provide a rich string operations set. I think there isn't a way to do global search-and-replace through some string (which can be a solution if we could replace all %2B with +, %2F with / and %3D with =). However there are circumstances under which nginx performs an urldecoding of some string - when this string becomes a part of an URI which will be forwarded to an upstream proxy server.
So we can add a value of a BASIC_AUTH request argument to the URI and make a proxy request to ourself:
# Main server block
server {
listen 80 default_server;
...
location / {
if ($arg_basic_auth) {
# "basic_auth" request argument is present,
# append "/decode_basic_auth/<BASE64_token>" to the URI
# and go to the next location block
rewrite ^(.*)$ /decode_basic_auth/$arg_basic_auth$1 last;
}
# No "basic_auth" request argument present,
# can do a proxy call from here without setting authorization headers
...
}
location /decode_basic_auth/ {
# This will be an internal location only
internal;
# Remove "basic_auth" request argument from the list of arguments
if ($args ~* (.*)(^|&)basic_auth=[^&]*(\2|$)&?(.*)) {
set $args $1$3$4;
}
# Some hostname for processing proxy subrequests
proxy_set_header Host internal.basic.auth.localhost;
# Do a subrequest to ourselfs, preserving other request arguments
proxy_pass http://127.0.0.1$uri$is_args$args;
}
}
# Additional server block for proxy subrequests processing
server {
listen 80;
server_name internal.basic.auth.localhost;
# Got URI in form "/decode_basic_auth/<BASE64_token>/<Original_URI>"
location ~ ^/decode_basic_auth/([^/]+)(/.*)$ {
proxy_set_header Authorization "Basic $1";
# Setup other HTTP headers here
...
proxy_pass http://<upstream_server>$2$is_args$args;
}
# Do not serve other requests
location / {
return 444;
}
}
Maybe this is not a very elegant solution, but it is tested and works.
OpenResty / ngx_http_lua_module
This can be easily solved with openresty or ngx_http_lua_module using ngx.escape_uri function:
server {
listen 80 default_server;
...
location / {
set $auth $arg_basic_auth;
if ($args ~* (.*)(^|&)basic_auth=[^&]*(\2|$)&?(.*)) {
set $args $1$3$4;
}
rewrite_by_lua_block {
ngx.var.auth = ngx.unescape_uri(ngx.var.auth)
}
proxy_set_header Authorization "Basic $auth";
# Setup other HTTP headers here
...
proxy_pass http://<upstream_server>;
}
}

how to get url parameter and pass to proxy_pass using nginx

I need to get the parameter from an URL, for example, abc=MY_STRING:
https://my-address/test?abc=MY_STRING
And at the reverse proxy (my-address), is configured like this:
location /test?(.*) {
proxy_pass http://local-server:1234/test?$args
}
but it is not working.
I tried another configuration:
location /test?(.*) {
proxy_pass http://local-server:1234/test?$1
}
but not worked too.
You cannot match the query string part of the URI with a location or rewrite statement, as it is not part of the normalized URI.
But you don't need to. The URI (complete with query string) will be passed upstream
unless you redirect it using a rewrite or try_files statement.
For example:
location /test {
proxy_pass http://localhost:1234;
}
The URI /test?abc=MY_STRING will match the location and be passed to localhost:1234 exactly the same. See this document for more.

Resources