How do I rewrite URLs in a proxy response in NGINX - nginx

I'm used to using Apache with mod_proxy_html, and am trying to achieve something similar with NGINX. The specific use case is that I have an admin UI running in Tomcat on port 8080 on a server at the root context:
http://localhost:8080/
I need to surface this on port 80, but I have other contexts on the NGINX server running on this host, so want to try and access this at:
http://localhost:80/admin/
I was hoping that the following super simple server block would do it, but it doesn't quite:
server {
listen 80;
server_name screenly.local.akana.com;
location /admin/ {
proxy_pass http://localhost:8080/;
}
}
The problem is that the returned content (html) contains URLs to scripts and style info that is all accessed at the root context, so I need to get these URLs rewritten to start with /admin/ instead of /.
How do I do this in NGINX?

We should first read the documentation on proxy_pass carefully and fully.
The URI passed to upstream server is determined based on whether "proxy_pass" directive is used with URI or not. Trailing slash in proxy_pass directive means that URI is present and equal to /. Absense of trailing slash means hat URI is absent.
Proxy_pass with URI:
location /some_dir/ {
proxy_pass http://some_server/;
}
With the above, there's the following proxy:
http:// your_server/some_dir/ some_subdir/some_file ->
http:// some_server/ some_subdir/some_file
Basically, /some_dir/ gets replaced by / to change the request path from /some_dir/some_subdir/some_file to /some_subdir/some_file.
Proxy_pass without URI:
location /some_dir/ {
proxy_pass http://some_server;
}
With the second (no trailing slash): the proxy goes like this:
http:// your_server /some_dir/some_subdir/some_file ->
http:// some_server /some_dir/some_subdir/some_file
Basically, the full original request path gets passed on without changes.
So, in your case, it seems you should just drop the trailing slash to get what you want.
Caveat
Note that automatic rewrite only works if you don't use variables in proxy_pass. If you use variables, you should do rewrite yourself:
location /some_dir/ {
rewrite /some_dir/(.*) /$1 break;
proxy_pass $upstream_server;
}
There are other cases where rewrite wouldn't work, that's why reading documentation is a must.
Edit
Reading your question again, it seems I may have missed that you just want to edit the html output.
For that, you can use the sub_filter directive. Something like ...
location /admin/ {
proxy_pass http://localhost:8080/;
sub_filter "http://your_server/" "http://your_server/admin/";
sub_filter_once off;
}
Basically, the string you want to replace and the replacement string

You may also need the following directive to be set before the first "sub_filter" for backend-servers with data compression:
proxy_set_header Accept-Encoding "";
Otherwise it may not work.
For your example it will look like:
location /admin/ {
proxy_pass http://localhost:8080/;
proxy_set_header Accept-Encoding "";
sub_filter "http://your_server/" "http://your_server/admin/";
sub_filter_once off;
}

You can use the following nginx configuration example:
upstream adminhost {
server adminhostname:8080;
}
server {
listen 80;
location ~ ^/admin/(.*)$ {
proxy_pass http://adminhost/$1$is_args$args;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
}

If there are no hyperlinks which need to be rewritten with sub_filter,
you might just use the proxy_redirect directive:
location /admin/ {
proxy_pass http://localhost:8080/;
proxy_redirect / /admin/
}
It changes the Location-Header of the response according to the given 'match-rewrite' rule.

Related

Sharing location configuration in nginx

I could not decide the best name for the question.
Essentially what I want to achieve is to set a custom allowed body size for a specific location on the webserver.
On the other hand, I was able to achieve the necessary result already with duplicate code, so I am really looking for a way how to make the code reusable and to better understand the observed behavior.
The server reverse-proxies all API requests to the backend service.
In global nginx config /etc/nginx/nginx.conf I set the rule for max allowed body size like so client_max_body_size 50k;.
Then, in individual server config /etc/nginx/conf.d/example.com I have the following config (simplified):
server {
listen 80;
listen [::]:80;
server_name api.example.com www.api.example.com
location ~* /file/upload {
client_max_body_size 100M;
# crashes without this line
proxy_pass http://localhost:90;
#proxy_pass http://localhost:90/file/upload; # also works
}
location / {
# does not work
#location ~* /file/upload {
# client_max_body_size 100M;
#}
proxy_pass http://localhost:90;
}
}
I am trying to override the max body size for file upload endpoint. See that there is 1 proxy_pass for location /file/upload and another proxy_pass for location / pointing to the same internal service.
Question 1. If I remove the proxy_pass from the location /file/upload then error is returned by the server. (no status code in chrome debugger). Why is this happening? Shouldn't request be propagated further to location /?
Question 2. Why is it not possible to define the sublocation with body size override inside the / location as in commented section in example above? If I set it like this, then 413 error code is returned, which hints that the client_max_body_size rule is ignored..
Question 3. Finally, is it possible to tell nginx, after the request hits the /file/upload location - to apply all the rules from the / section? I guess one solution to this problem would be to move the common configuration into separate file and then import it in both sections.. I was thinking if there is any solution that does not require creating new files?
Here is the reusable config I am talking about basically:
location / {
#.s. kill cache. use in dev
sendfile off;
# kill cache
add_header Last-Modified $date_gmt;
add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
if_modified_since off;
expires off;
etag off;
# don't cache it
proxy_no_cache 1;
# even if cached, don't try to use it
proxy_cache_bypass 1;
proxy_pass http://localhost:90;
client_max_body_size 100M;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass_request_headers on;
}
This is not the final version. If I had to copy this piece of code to 2 sections this would not be very friendly approach. So, it would be nice to hear some nginx lifehacks on how accomplish what I try to accomplish in a most friendly way and get some explanations for observed behavior.
Answer 1
If I remove the proxy_pass from the location /file/upload then error is returned by the server. (no status code in chrome debugger). Why is this happening?
Every location have a so-called content handler. If you don't specify content handler explicitly via proxy_pass (fastcgi_pass, uwsgi_pass, etc.) directive, nginx will try to serve the request locally.
Shouldn't request be propagated further to location /?
Of course not. What makes you think it should?
Answer 2
Why is it not possible to define the sublocation with body size override inside the / location as in commented section in example above? If I set it like this, then 413 error code is returned, which hints that the client_max_body_size rule is ignored..
I'd rather expect you'll get the same error as in the first case since your nested location does not have an explicitly specified content handler via the proxy_pass directive. However the following config is worth to try:
location / {
# all the common configuration
location /file/upload {
client_max_body_size 100M;
proxy_pass http://localhost:90;
}
proxy_pass http://localhost:90;
}
Answer 3
Finally, is it possible to tell nginx, after the request hits the /file/upload location - to apply all the rules from the / section?
No, unless you use a separate file via include directive in both locations. However you can try to move all the upstream related setup directives one level up to the server context:
server {
...
# all the common configuration
location / {
proxy_pass http://localhost:90;
}
location /file/upload {
client_max_body_size 100M;
proxy_pass http://localhost:90;
}
}
Note that some directives (e.g. add_header, proxy_set_header) are inherited from the previous configuration level if and only if there are no those directives defined on the current level.
Very often dynamic settings for different locations can be achieved using the map block in a following way:
map $uri $max_body_size {
~^/file/upload 100M;
default 50k;
}
server {
location / {
...
client_max_body_size $max_body_size;
...
}
}
Unfortunally not every nginx directive accepts variable as its argument. Usually when nginx documentation doesn't explicitly states that some directive can accept variables, it means it cannot, and the client_max_body_size is exactly that kind of directive, so the above configuration won't work.

Nginx set browser location root [duplicate]

I'm used to using Apache with mod_proxy_html, and am trying to achieve something similar with NGINX. The specific use case is that I have an admin UI running in Tomcat on port 8080 on a server at the root context:
http://localhost:8080/
I need to surface this on port 80, but I have other contexts on the NGINX server running on this host, so want to try and access this at:
http://localhost:80/admin/
I was hoping that the following super simple server block would do it, but it doesn't quite:
server {
listen 80;
server_name screenly.local.akana.com;
location /admin/ {
proxy_pass http://localhost:8080/;
}
}
The problem is that the returned content (html) contains URLs to scripts and style info that is all accessed at the root context, so I need to get these URLs rewritten to start with /admin/ instead of /.
How do I do this in NGINX?
We should first read the documentation on proxy_pass carefully and fully.
The URI passed to upstream server is determined based on whether "proxy_pass" directive is used with URI or not. Trailing slash in proxy_pass directive means that URI is present and equal to /. Absense of trailing slash means hat URI is absent.
Proxy_pass with URI:
location /some_dir/ {
proxy_pass http://some_server/;
}
With the above, there's the following proxy:
http:// your_server/some_dir/ some_subdir/some_file ->
http:// some_server/ some_subdir/some_file
Basically, /some_dir/ gets replaced by / to change the request path from /some_dir/some_subdir/some_file to /some_subdir/some_file.
Proxy_pass without URI:
location /some_dir/ {
proxy_pass http://some_server;
}
With the second (no trailing slash): the proxy goes like this:
http:// your_server /some_dir/some_subdir/some_file ->
http:// some_server /some_dir/some_subdir/some_file
Basically, the full original request path gets passed on without changes.
So, in your case, it seems you should just drop the trailing slash to get what you want.
Caveat
Note that automatic rewrite only works if you don't use variables in proxy_pass. If you use variables, you should do rewrite yourself:
location /some_dir/ {
rewrite /some_dir/(.*) /$1 break;
proxy_pass $upstream_server;
}
There are other cases where rewrite wouldn't work, that's why reading documentation is a must.
Edit
Reading your question again, it seems I may have missed that you just want to edit the html output.
For that, you can use the sub_filter directive. Something like ...
location /admin/ {
proxy_pass http://localhost:8080/;
sub_filter "http://your_server/" "http://your_server/admin/";
sub_filter_once off;
}
Basically, the string you want to replace and the replacement string
You may also need the following directive to be set before the first "sub_filter" for backend-servers with data compression:
proxy_set_header Accept-Encoding "";
Otherwise it may not work.
For your example it will look like:
location /admin/ {
proxy_pass http://localhost:8080/;
proxy_set_header Accept-Encoding "";
sub_filter "http://your_server/" "http://your_server/admin/";
sub_filter_once off;
}
You can use the following nginx configuration example:
upstream adminhost {
server adminhostname:8080;
}
server {
listen 80;
location ~ ^/admin/(.*)$ {
proxy_pass http://adminhost/$1$is_args$args;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
}
If there are no hyperlinks which need to be rewritten with sub_filter,
you might just use the proxy_redirect directive:
location /admin/ {
proxy_pass http://localhost:8080/;
proxy_redirect / /admin/
}
It changes the Location-Header of the response according to the given 'match-rewrite' rule.

nginx proxy config not forwarding requests to backend server

Below is the relevant section of my nginx.conf file.
I only see the js|css... requests forward to my backend server when i remove the initial location block in the conf file. What im trying to accomplish is to turn off nginx access logging for files of those extensions.
Anybody know a working nginx config technique to allow me to turn off the access logs yet still forward these requests to the proxy location?
...
location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
access_log off;
}
location / {
if ($ignore_ua) {
access_log off;
return 200;
}
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:7777/;
}
nginx chooses a location block to process a request. In the case of .js files, your location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ block is used. None of the directives within the location / block are involved. See this document for details.
If you need conditional logging, you could use the if= parameter to the access_log directive instead of a separate location block. See this document for an example.
In your case, it might look like this:
map $request_uri $loggable {
default 1;
\.(js|css|png|jpg|jpeg|gif|ico)(\?|$) 0;
}
access_log /path/to/access.log combined if=$loggable;
Note that the map directive goes in the http block.

NGINX reverse proxy with Rewrites to replace TMG2010

I curently have a Forefront TMG2010 doing reverse proxy to several internal web servers. Since it was really easy in TMG I have done lots of host and virtual directory mappings/rewrites.
Now I ened to acheive the same in NGINX.
e.g.,
apphost.domain.com/ needs to map internally to server1/apppath/
apphost.domain.com/test/ needs to map internally to server2/apppath/
apphost.domain.com/dev/ needs to map internally to server3/apppath/
in each case the internal /apppath/ needs to be invisible to the client. This is to achieve the same as the external to internal path mapping in TMG.
What I have done so far is;
server {
listen 80;
server_name host.server.com;
location /test/ {
proxy_set_header Host $host;
rewrite ^(.*)$ /app/$1 break;
proxy_pass http://10.0.0.2;
}
location / {
proxy_set_header Host $host;
rewrite ^(.*)$ /app/$1 break;
proxy_pass http://10.0.0.1;
}
But only the / seems to work, the /test/ doesnt.
Help!
So you want to map /test/foo to http://10.0.0.2/app/foo. The simplest solution is:
location /test/ {
proxy_set_header Host $host;
proxy_pass http://10.0.0.2/app/;
}
According to the documentation the /test/ element will be replaced by /app/ as it passes upstream.

Nginx pass_proxy subdirectory without url decoding

I need to write an nginx location directive to proxy requests to subdirectory to another server preserving urlencoding and removing subdirectory prefix.
Here's an artificial example — request like this:
http://1.2.3.4/api/save/http%3A%2F%2Fexample.com
should pass as
http://abcd.com/save/http%3A%2F%2Fexample.com
I tried several different ways. Here're couple of them:
From this SO question
location /api/ {
rewrite ^/api(/.*) $1 break;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_pass http://abcd.com;
}
But it decodes the string, so http://abcd.com gets /save/http://example.com
From another SO question
location /api/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_pass http://abcd.com;
}
But it keeps subdirectory, so http://abcd.com gets /api/save/http%3A%2F%2Fexample.com.
What's needed is somewhere in the middle. Thank you!
UPD: Here's a ticket in nginx bug tracker
But there is no easy way to fix this nginx behaviour. There are some bugs in nginx trac, you could add yours. trac.nginx.org/nginx/…. So, I think that the simplest way is to have subdomain. – Alexey Ten Feb 24 '15 at 14:49
https://trac.nginx.org/nginx/ticket/727
If you want nginx to do something custom, you can do so using ​proxy_pass with variables (and the $request_uri variable, which contains original unescaped request URI as sent by a client). In this case it will be your responsibility to do correct URI transformations. Note though that this can easily cause security issues and should be done with care.
Challenge accepted!
location /api/ {
rewrite ^ $request_uri;
rewrite ^/api/(.*) $1 break;
return 400;
proxy_pass http://127.0.0.1:82/$uri;
}
That's it, folks!
Here's for the full proof.
The config file for nginx/1.2.1:
server {
listen 81;
#first, the solution
location /api/ {
rewrite ^ $request_uri;
rewrite ^/api/(.*) $1 break;
return 400; #if the second rewrite won't match
proxy_pass http://127.0.0.1:82/$uri;
}
#next, a few control groups
location /dec/ {
proxy_pass http://127.0.0.1:82/;
}
location /mec/ {
rewrite ^/mec(/.*) $1 break;
proxy_pass http://127.0.0.1:82;
}
location /nod/ {
proxy_pass http://127.0.0.1:82;
}
}
server {
listen 82;
return 200 $request_uri\n;
}
Here are the results of running the queries for each location:
% echo localhost:81/{api,dec,mec,nod}/save/http%3A%2F%2Fexample.com | xargs -n1 curl
/save/http%3A%2F%2Fexample.com
/save/http:/example.com
/save/http:/example.com
/nod/save/http%3A%2F%2Fexample.com
%
Note that having that extra return 400; is quite important — otherwise, you risk having a security issue (file access through //api etc), as Maxim has briefly mentioned in your trac ticket.
P.S. If you think using the rewrite engine as a finite-state automaton is super cool, you might also want check out my http://mdoc.su/ project, or fork it github.
What you have to do is fairly easy as long as we are talking prefix matching with ^~ or no modifier
location /api/ {
# if you don't want to pass /api/ add a trailing slash to the proxy_pass
proxy_pass http://localhost:8080/;
...
}
And everything will be passed along without decoding, you don't have to pass $uri
Also while you use proxy pass you should also set these headers
# pass headers and body along
proxy_pass_request_headers on;
proxy_pass_request_body on;
# set some headers to make sure the reverse proxy is passing along everything necessary
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Resources