I am looking for a way to do the following using Nginx:
Intercept a request
Read URL, parse it and read a value from it.
Add that value as a new request header
Update the URL (remove a particular value)
Forward the request to another server
e.g
Request URL - http://<<nginx>>/test/001.xml/25
Final URL - http://<<server>>/test/001.xml with header (x-replica: 25)
I have a nginx server setup with a upstream for the actual server. I was wondering how do I setup Nginx to achieve this ?
Since the data exists within the request URI itself (available by the $uri variable in nginx), you can parse that using the nginx lua module. nginx will need to be compiled with lua for this to work, see: openresty's nginx lua module.
From there you can use the set_by_lua_block or set_by_lua_file directive given $uri as a parameter.
In configuration this would look something like:
location / {
...
set_by_lua_file $var_to_set /path/to/script.lua $uri;
# $var_to_set would contain the result of the script from this point
proxy_set_header X-Replica $var_to_set;
...
}
In script.lua we can access the $uri variable from in the ngx.arg list (see these docs):
function parse_uri( uri )
parsed_uri = uri
-- Parse logic here
return parsed_uri
end
return parse_uri( ngx.arg[1] )
Similarly, you can modify this function or create another to make a variable with the updated $uri.
Related
I've got a particular scenario where I'm needing to route to a different backend based on query parameter:
https://edge1.cdn.com/file.zip?{secure_link}&{tokens}&route=aws1
Where aws1 would be say http://edge1.amazonwebservices.com
and if its aws2 then proxy backend would be http://edge2.amazonwebservices.com
and so on... but I still have not figured out how to do this.
You can use map directive to get a proxy hostname from the $arg_route variable (which contains a value of the route query argument):
map $arg_route $aws {
aws1 edge1.amazonwebservices.com;
aws2 edge2.amazonwebservices.com;
...
default <default_hostname>;
}
server {
...
# if you want to proxy the request, you'd need a 'resolver' directive
resolver <some_working_DNS_server_address>;
location / {
# if you want to proxy the request
proxy_pass http://$aws;
# or if you want to redirect the request
rewrite ^ http://$aws$uri permanent;
}
}
If you don't want to serve the request without route query argument, you can omit the last default line at the map block and add the following if block to your server configuration:
if ($aws = '') {
return 403; # HTTP 403 denied
}
If you need to proxy the request you'd additionally need a resolver directive (you can read some technical details about it in this article).
I have an Nginx instance running as a reverse proxy. When the upstream server does not respond, I send a custom error page for the 502 response code. When the upstream server sends an error page, that gets forwarded to the client, and I'd like to show a custom error page in that case as well.
If I wanted to replace all of the error pages from the upstream server, I would set proxy_intercept_errors on to show a custom page on each of them. However, there are cases where I'd like to return the actual response that the upstream server sent: for example, for API endpoints, or if the error page has specific user-readable text relating to the issue.
In the config, a single server is proxying multiple applications that are behind their own proxy setups and their own rules for forwarding requests around, so I can't just specify this per each location, and it has to work for any URL that matches a server.
Because of this, I would like to send the custom error page, unless the upstream application says not to. The easiest way to do this would be with a custom HTTP header. There is a similar question about doing this depending on the request headers. Is there a way to do this depending on the response headers?
(It appears that somebody else already had this question and their conclusion was that it was impossible with plain Nginx. If that's true, I would be interested in some other ideas on how to solve this, possibly using OpenResty like that person did.)
So far I have tried using OpenResty to do this, but it doesn't seem compatible with proxy_pass: the response that the Lua code generates seems to overwrite the response from the upstream server.
Here's the location block I tried to use:
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:65000;
content_by_lua_block{
ngx.say("This seems to overwrite the content from the proxy?!")
}
body_filter_by_lua_block {
ngx.arg[1]="Truncated by code!"
ngx.arg[2]=false
if ngx.status >= 400 then
if not ngx.resp.get_headers()["X-Verbatim"] then
local file = io.open('/usr/share/nginx/error.html', 'w')
local html_text = file:read("*a")
ngx.arg[1] = html_text
ngx.arg[2] = true
return
end
end
}
}
I don't think that you can send custom error pages based on the response header since the only way, as per my knowledge, you could achieve that was using either map or if directive. Since both of these directives don't have scope after the request is sent to the upstream, they can't possibly read the response header.
However, you could do this using openresty and writing your own lua script. The lua script to do such a thing would look something like:
location / {
body_filter_by_lua '
if ngx.resp.get_headers()["Cust-Resp-Header"] then
local file = io.open('/path/to/file.html', 'r')
local html_text = f:read()
ngx.arg[1] = html_text
ngx.arg[2] = true
return
end
';
#
.
.
.
}
You could also use body_filter_by_lua_block (you could enclose your lua code inside curly brances instead writing as nginx string) or body_filter_by_lua_file (you could write your lua code in a separate file and provide the file path).
You can find how to get started with openresty here.
P.S.: You can read the response status code from the upstream using ngx.status. As far as reading the body is concerned, the variable ngx.arg[1] would contain the response body after the response from the upstream which we're modifying here. You can save the ngx.arg[1] in a local variable and try to read the error message from that using some regexp and appending later in the html_text variable. Hope that helps.
Edit 1: Pasting here a sample working lua block inside a location block with proxy_pass:
location /hello {
proxy_pass http://localhost:3102/;
body_filter_by_lua_block {
if ngx.resp.get_headers()["erratic"] == "true" then
ngx.arg[1] = "<html><body>Hi</body></html>"
end
}
}
Edit 2: You can't use content_by_lua_block with proxy_pass or else your proxy wouldn't work. Your location block should look like this (assuming X-Verbatim header is set to "false" (a string) if you've to override the error response body from the upstream).
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:65000;
body_filter_by_lua_block {
if ngx.status >= 400 then
if ngx.resp.get_headers()["X-Verbatim"] == "false" then
local file = io.open('/usr/share/nginx/error.html', 'w')
local html_text = file:read("*a")
ngx.arg[1] = html_text
ngx.arg[2] = true
end
end
}
}
This is somewhat opposite of the requested but I think it can fit anyway. It shows the original response unless upstream says what to show.
There is a set of X-Accel custom headers that are evaluated from upstream responses. X-Accel-Redirect allows you to tell NGINX to process another location instead. Below is an example how it can be used.
This is a Flask application that gives 50/50 normal responses and errors. The error responses come with X-Accel-Redirect header, instructing NGINX to reply with contents from the #error_page location.
import flask
import random
application = flask.Flask(__name__)
#application.route("/")
def main():
if random.randint(0, 1):
resp = flask.Response("Random error") # upstream body contents
resp.headers['X-Accel-Redirect'] = '#error_page' # the header
return resp
else:
return "Normal response"
if __name__ == '__main__':
application.run("0.0.0.0", port=4000)
And here is a NGINX config for that:
server {
listen 80;
location / {
proxy_pass http://localhost:4000/;
}
location #error_page {
return 200 "That was an error";
}
}
Putting these together you will see either "Normal response" from the app, or "That was an error" from the #error_page location ("Random error" will be suppressed). With this setup you can create a number of various locations (#error_502, #foo, #etc) for various errors and make your application to use them.
I have a lua script that uses lua-resty to call another service via co-sockets.
Now I would like to use the information from this call to route the request in nginx.
nginx includes the lua script in access_by_lua*
which sets the var like this:
ngx.var.blocked = '1'
and routes in the location like this:
if ( $blocked ) {
proxy_pass http://haproxy-9001;
break;
}
the problem is now that nginx does not pick up the variable change (in this phase).
if I include the lua script in set_by_lua* phase then the variable passing works but I dont have access to the co-sockets in this phase.
Any idea how to get the variable out of lua land into the nginx variable in the access_by_lua, rewrite_by_lua or content_by_lua phase so that I can use the co-socket api to make a http call?
if nginx directive is implemented by https://nginx.ru/en/docs/http/ngx_http_rewrite_module.html.
Obviously it works on rewrite phase, so your changes at access phase doesn't work.
Just don't use if. Below is snippet from one of my nginx config:
proxy_pass $scheme://$upstream$uri$is_args$args;
Just set $upstream variable at access phase and it will work at content phase (proxy_pass).
Maybe you could capture location with that proxy instead of variable, it works in access_by_lua scope
https://github.com/openresty/lua-nginx-module#ngxlocationcapture
I'm using Nginx (version 1.9.9) as a reverse proxy to my backend server. It needs to perform authentication/authorization based on the contents of the POST requests. And I'm having trouble reading the POST request body in my auth_request handler. Here's what I got.
Nginx configuration (relevant part):
server {
location / {
auth_request /auth-proxy;
proxy_pass http://backend/;
}
location = /auth-proxy {
internal;
proxy_pass http://auth-server/;
proxy_pass_request_body on;
proxy_no_cache "1";
}
}
And in my auth-server code (Python 2.7), I try to read the request body like this:
class AuthHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def get_request_body(self):
content_len = int(self.headers.getheader('content-length', 0))
content = self.rfile.read(content_len)
return content
I printed out the content_len and it had the correct value. However, the self.rfile.read() will simply hang. And eventually it will time out and returns "[Errno 32] Broken pipe".
This is how I posted test data to the server:
$ curl --data '12345678' localhost:1234
The above command hangs as well and eventually times out and prints "Closing connection 0".
Any obvious mistakes in what I'm doing?
Thanks much!
The code of the nginx-auth-request-module is annotated at nginx.com. The module always replaces the POST body with an empty buffer.
In one of the tutorials, they explain the reason, stating:
As the request body is discarded for authentication subrequests, you will
need to set the proxy_pass_request_body directive to off and also set the
Content-Length header to a null string
The reason for this is that auth subrequests are sent at HTTP GET methods, not POST. Since GET has no body, the body is discarded. The only workaround with the existing module would be to pull the needed information from the request body and put it into an HTTP header that is passed to the auth service.
We know that HAProxy and Nginx can do the URL hash based upstream but how can we hash the part of the URL.
We have 4 back-end original image servers, each will store all the original large size image files. The image server will resize the file based on user request on the fly. (Tomcat Java load the file into memory and resize then response)
The original file is:
http://imageserver.company.com/path/to/imageA.jpg
The end-user will request:
httpurl://imageserver.company.com/path/to/imageA.jpg/crop/400x300.jpg
httpurl://imageserver.company.com/path/to/imageA.jpg/400x224.jpg
httpurl://imageserver.company.com/path/to/imageA.jpg/1280x720.jpg
I would like HAProxy and Nginx will do the hash on "/path/to/imageA.jpg";
Hash (substring (url, 0, find (url, ".jpg/")
Any idea of how to config?
In nginx you can use the map and upstream::hash directives:
map $uri $image_hash {
default $uri;
"~(?<image_path>.+(?:jpg|png))/" $image_path;
}
upstream image_backends {
hash $image_hash;
server server1;
server server2;
server server3;
server server4;
}
server {
...
location / {
# add debug header to view the hash
add_header ImageHash $image_hash;
proxy_pass http://image_backends;
}
}
I'm not sure what the exact syntax would be for HAProxy, but it's uri hash supports specifying the "depth" of the URI hash. So if the original path to the URL has a fixed depth then you can use that (though I'm guessing that's not the case)?
The "depth" parameter indicates the maximum directory depth
to be used to compute the hash. One level is counted for each
slash in the request. If both parameters are specified, the
evaluation stops when either is reached.