Using Lua in nginx to pass a request to FastCGI - nginx

Using nginx compiled with Lua support, how can we make a sort of sub-request to a FastCGI handler, much like nginx's fastcgi_pass directive?
What I'd like to do is something like this:
location = / {
access_by_lua '
res = ngx_fastcgi.pass("backend")
';
}
(Obviously, this doesn't work.)
I'm pouring over HttpLuaModule where I see mention ngx_fastcgi and ngx.location.capture, which, evidently, makes
non-blocking internal requests to other locations configured with
disk file directory or any other nginx C modules like ... ngx_fastcgi,
...
But then following the link of ngx_fastcgi takes me to HttpFastcgiModule which explains only nginx directives, not Lua-scriptable commands. Is ngx.location.capture the right function to use? (These requests, by the way, will be to localhost, just on a different port, like 9000 or 9001.)
How can I use Lua in nginx to forward a request, or make a sub-request, to a FastCGI endpoint?

Use the ngx.location.capture() method to perform a subrequest to a predefined location block. Then, from within the location block, perform the external, FastCGI request. Because the subrequest itself isn't actually a network operation, but is performed purely within nginx C-based environment, there's very little overhead. Further, because the FastCGI request and other "proxy_pass"-type requests are event-based, nginx can operate as an efficient intermediary.
As an example, you could have the following:
location / {
access_by_lua '
response = ngx.location.capture("/my-subrequest-handler")
if response.status == 404 then
return ngx.exit(401) -- can't find/authenticate user, refuse request
end
ngx.say(response.status)
';
# other nginx config stuff here as necessary--perhaps another fastcgi_pass
# depending upon the status code of the response above...
}
location = /my-subrequest-handler {
internal; # this location block can only be seen by nginx subrequests
fastcgi_pass localhost:9000; # or some named "upstream"
fastcgi_pass_request_body off; # send client request body upstream?
fastcgi_pass_request_headers off; # send client request headers upstream?
fastcgi_connect_timeout 100ms; # optional; control backend timeouts
fastcgi_send_timeout 100ms; # same
fastcgi_read_timeout 100ms; # same
fastcgi_keep_conn on; # keep request alive
include fastcgi_params;
}
In the above example, even though I'm performing a subrequest to "/my-subrequest-handler", the actual URL passed to the FastCGI process is the one requested by the HTTP client calling into nginx in the first place.
Note that that ngx.location.capture is a synchronous, but non-blocking operation which means that your code execution stops until a response is received, but the nginx worker is free to perform other operations in the meantime.
There are some really cool things that you can do with Lua to modify the request and response at any point in the nginx pipeline. For example, you could change the original request by adding headers, removing headers, even transforming the body. Perhaps the caller wants to work with XML, but the upstream application only understands JSON, we can convert to/from JSON when calling the upstream application.
Lua is not built into nginx by default. Instead it's a 3rd party module that must be compiled in. There's a flavor of nginx called OpenResty that builds in Lua+LuaJIT along with a few other modules that you may or may not need.

Related

What's the best way to configure LRU cache using open-resty lua scripts in nginx proxy

I am using openresty lua (https://github.com/openresty) to configure our nginx proxy. I have one main proxy.template that defines 3 locations, but want to use only one LRU cache for multiple process initialization calls (since they take so long and nothing else, that is why I am using a script). I want to define a variable that can be passed into each location, but am pretty sure I am not doing this correctly. I have:
#init_by_lua_file $lru_cache /etc/scripts/lua/process_cache.lua;
location /process {
access_by_lua_file /etc/scripts/lua/process_access.lua;
proxy_set_header Content-Type "application/json";
proxy_set_header Accept "application/json";
proxy_ssl_server_name on;
proxy_pass $target;
}
location /process/init {
set_by_lua_file $lru_cache /etc/scripts/lua/process_cache.lua;
add_header Access-Control-Expose-Headers set-cookie;
add_header Access-Control-Allow-Headers set-cookie;
access_by_lua_file /etc/scripts/lua/process_init.lua;
}
The process_cache creates the cache (one per proxy startup) and I would like it to be referenced by the process_init.lua and process_access.lua which do different things. For example, process_init is only called once for a UI initialization and establishes the specific cache entries, process_access checks to make sure the entry hasn't expired and if not uses it, otherwise creates a new entry, so that a long call to another server is not needed.
The above would require the lru_cache variable to be passed amongst the two locations. My latest attempts were in the area of trying to place process_cache.lua within the /process/init path, but then it just gets initialized each time, so starting with an empty cache each /process/init call is useless. Thoughts?
consider ngx.shared ?
You can specific exptime for your strings.
: https://github.com/openresty/lua-nginx-module#ngxshareddict ?

Avoid redirect cycle when combining access_by_lua with ngx.exec to a (named) location

I want to change an existing nginx configuration in a way where I can completely "mask" the configuration and proxy everything to the upstream when a certain cookie is available (to hide a certain server).
This includes not just some location directives but basically every location directive (as opposed to set or map a variable and update n-location's try_files and more).
My basic idea was to use lua and jump into the Rewrite/Access phase like this:
access_by_lua_block {
# proceed as usual if our cookie is not detected
if ngx.var.cookie_demo ~= nil and string.len(ngx.var.cookie_demo) ~= 32 then
return
end
# proxy and return w/out further processing
ngx.exec("#ngxbackend")
return ngx.exit(ngx.HTTP_OK)
}
# proxy upstream
location #ngxbackend {
include /etc/nginx/proxy_params_demo;
proxy_pass https://demo-upstreams;
}
But this leads to an ERR with rewrite or internal redirection cycle while redirect to named location "#ngxbackend" as the named location is probably never reached because of the access_by_lua_block after it's internal redirect.
Can I solve this by use of variables and further condition checking?

Nginx auth_request handler accessing POST request body?

I'm using Nginx (version 1.9.9) as a reverse proxy to my backend server. It needs to perform authentication/authorization based on the contents of the POST requests. And I'm having trouble reading the POST request body in my auth_request handler. Here's what I got.
Nginx configuration (relevant part):
server {
location / {
auth_request /auth-proxy;
proxy_pass http://backend/;
}
location = /auth-proxy {
internal;
proxy_pass http://auth-server/;
proxy_pass_request_body on;
proxy_no_cache "1";
}
}
And in my auth-server code (Python 2.7), I try to read the request body like this:
class AuthHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def get_request_body(self):
content_len = int(self.headers.getheader('content-length', 0))
content = self.rfile.read(content_len)
return content
I printed out the content_len and it had the correct value. However, the self.rfile.read() will simply hang. And eventually it will time out and returns "[Errno 32] Broken pipe".
This is how I posted test data to the server:
$ curl --data '12345678' localhost:1234
The above command hangs as well and eventually times out and prints "Closing connection 0".
Any obvious mistakes in what I'm doing?
Thanks much!
The code of the nginx-auth-request-module is annotated at nginx.com. The module always replaces the POST body with an empty buffer.
In one of the tutorials, they explain the reason, stating:
As the request body is discarded for authentication subrequests, you will
need to set the proxy_pass_request_body directive to off and also set the
Content-Length header to a null string
The reason for this is that auth subrequests are sent at HTTP GET methods, not POST. Since GET has no body, the body is discarded. The only workaround with the existing module would be to pull the needed information from the request body and put it into an HTTP header that is passed to the auth service.

Proxy a request - get a parameter from URL, add a header and update request URL using Nginx

I am looking for a way to do the following using Nginx:
Intercept a request
Read URL, parse it and read a value from it.
Add that value as a new request header
Update the URL (remove a particular value)
Forward the request to another server
e.g
Request URL - http://<<nginx>>/test/001.xml/25
Final URL - http://<<server>>/test/001.xml with header (x-replica: 25)
I have a nginx server setup with a upstream for the actual server. I was wondering how do I setup Nginx to achieve this ?
Since the data exists within the request URI itself (available by the $uri variable in nginx), you can parse that using the nginx lua module. nginx will need to be compiled with lua for this to work, see: openresty's nginx lua module.
From there you can use the set_by_lua_block or set_by_lua_file directive given $uri as a parameter.
In configuration this would look something like:
location / {
...
set_by_lua_file $var_to_set /path/to/script.lua $uri;
# $var_to_set would contain the result of the script from this point
proxy_set_header X-Replica $var_to_set;
...
}
In script.lua we can access the $uri variable from in the ngx.arg list (see these docs):
function parse_uri( uri )
parsed_uri = uri
-- Parse logic here
return parsed_uri
end
return parse_uri( ngx.arg[1] )
Similarly, you can modify this function or create another to make a variable with the updated $uri.

Message "X-Accel-Mapping header missing" in Nginx error log

I am running a Rails 3 site on Ubuntu 8.04 with Nginx 1.0.0 and Passenger 3.0.7.
In my Nginx error.log I started seeing the message X-Accel-Mapping header missing quite a lot. Googling lead me to the docs of Rack::Sendfile and to the Nginx docs.
Now, my app can be accessed through several domains and I am using send_file in my app to deliver some files specific to the domain they are requested from, e.g., if you come to domain1.com/favicon.ico I look up the favicon in at public/websites/domain1/favicon.ico.
This works fine and I don't think I need/want to get Nginx involved and create some private area where I store those files, as the samples in the Rack::Sendfile docs suggest.
How can I get rid of the error message?
this message means that Rack::Sendfile disabled X-Accel-Redirect for you, because you have missing configuration for it in nginx.conf...
I'm using Nginx + Passenger 3 + Rails 3.1.
Gathered information from this pages I've figured it out:
http://wiki.nginx.org/X-accel
http://greenlegos.wordpress.com/2011/09/12/sending-files-with-nginx-x-accel-redirect
http://code.google.com/p/substruct/source/browse/trunk/gems/rack-1.1.0/lib/rack/sendfile.rb?r=355
Serving Large Files Through Nginx via Rails 2.3 Using x-sendfile
I have controller which maps /download/1 requests to storage files which have their own directory structure, like this: storage/00/00/1, storage/01/0f/15 etc. So I need to pass this through Rails, but then I need to use send_file method which will use X-Accel-Redirect to send the final file to the browser through nginx directly.
Within the code I have this:
send_file(
'/var/www/shared/storage/00/00/01',
:disposition => :inline,
:filename => #file.name # an absolute path to the file which you want to send
)
I replaced the filename for this example purposes
Now I had to add these lines to my nginx.conf:
server {
# ...
passenger_set_cgi_param HTTP_X_ACCEL_MAPPING /var/www/shared/storage/=/storage/;
passenger_pass_header X-Accel-Redirect;
location /storage {
root /var/www/shared;
internal;
}
# ...
}
The path /storage is not visible from outside world, it is internal only.
Rack::Sendfile gets the header X-Accel-Mapping, extracts the path from it and replaces /var/www/shared/storage with /storage.... Then it spits out the modified header:
X-Accel-Redirect: /storage/00/00/01
which is then processed by nginx.
I can see this works correctly as the file is downloaded 100x faster than before and no error is shown in the logs.
Hope this helps.
We used the similar technique as NoICE described, but i replaced the "hard-coded" directory containing all the files with the regular expression describing the folder containing the folders containing the files.
Sounds hard, yeah? Just take a look on these (/etc/nginx/sites-available/my.web.site):
location /assets/(.+-[a-z0-9]+\.\w+) {
root /home/user/my.web.site/public/assets/$1;
internal;
}
location /images/(.+)(\?.*)? {
root /home/user/my.web.site/public/images/$1;
internal;
}
This should be used with this check:
location / {
# ...
if (-f $request_filename) {
expires max;
break;
}
# ...
}
to prevent the statics from Rails processing.
I did by this manual
https://mattbrictson.com/accelerated-rails-downloads
my server sends file path /private_upload/file/123/myfile.txt, the file is in /data/myapp-data/private_upload/file/123/myfile.txt
# Allow NGINX to serve any file in /data/myapp-data/private_upload
# via a special internal-only location.
location /private_upload {
internal;
alias /data/myapp-data/private_upload;
}
# ---------- BACKEND ----------
location #backend
{
limit_req zone=backend_req_limit_per_ip burst=20 nodelay;
proxy_pass http://backend;
proxy_set_header X-Sendfile-Type X-Accel-Redirect;
proxy_set_header X-Accel-Mapping /=/; # this header is required, it does nothing
include /etc/nginx/templates/myapp_proxy.conf;
}

Resources