How to write only logs with 200 status - nginx

I'm trying to figure out how to do the following:
Request is coming in.
HttpLuaModule performs some action against the request. If request is valid than Lua will finish processing with ngx.exit(202). But there are some conditions that may (and will) occur during the processing and nginx might return 403 , 404, 503 Errors.
What I want to do is to write to access logs only requests that have 200 Status code.
Basically I would like to do something like this:
location /foo {
content_by_lua_file "/opt/nginx/lua/process.lua";
if (status == 200) {
access_log "/path/to/the/access_log"
}
I'm very new to both nginx and lua so for me it's a bit of a challenge to figure out where to place and if statement (ether after content_by_lua_file or in side lua file) and what this if statement should look like.

nginx 1.7.0+ allows using an if condition in access_log directive itself.
access_log path [format [buffer=size [flush=time]] [if=condition]];
The if parameter (1.7.0) enables conditional logging.
A request will not be logged if the condition evaluates to “0” or an empty string
Combined with map directive its possible to send log events to different logs based on various conditions.
http {
map $status $normal {
~^2 1;
default 0;
}
map $status $abnormal {
~^2 0;
default 1;
}
map $remote_addr $islocal {
~^127 1;
default 0;
}
server {
access_log logs/access.log combined if=$normal;
access_log logs/access_abnormal.log combined if=$abnormal;
access_log logs/access_local.log combined if=$islocal;
}
}
http://nginx.org/en/docs/http/ngx_http_log_module.html
http://nginx.org/en/docs/http/ngx_http_map_module.html

you can do it by using ngx.log and log_by_lua directives.
location /conditional_log{
log_by_lua 'if ngx.status == 200 then ngx.log(ngx.ERR, "It is 200") end';
content_by_lua 'ngx.say("I am ok") ngx.exit(200)';
}
In the above code, we use log_by_lua which is called while running in log phase. In that if ngx.status == 200, we use ngx.log to trigger the logging using ngx.log.
This will write to error_log. Not sure how to write it to access_log.
For reference
http://wiki.nginx.org/HttpLuaModule#ngx.log
http://wiki.nginx.org/HttpLuaModule#log_by_lua

In every question is a part of answer. You were very close:
if ($status != "200") {
access_log off;
}
Check info for version availability here.
http://nginx.org/en/docs/http/ngx_http_core_module.html#variables
Also, almost all access log format vars are available in "modern" versions:
http://nginx.org/en/docs/http/ngx_http_log_module.html

This is the solution I came up with:
auth.lua
-- Some logic goes here
-- ....
-- ....
ngx.var.return_status = 200
nginx.conf
http {
lua_package_path .....;
lua_package_cpath ....;
rewrite_by_lua_no_postpone on;
server {
set $return_status 1;
location /foo {
rewrite_by_lua_file "<apth_to_aut.lua";
if ($return_status = 200) {
access_log <path_to_access_log> format;
return 200;
}
}
}
}

Related

How to get nginx to do a redirect to url-encoded query parameter

I have a requirement to do a proxy call to url delivered via a query parameter as per example:
My nginx proxy is deployed at: https://myproxy.net
if the redirect parameter is not url encoded I can do the call with this block:
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass $arg_redirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
the error intercepts and #handle_redirects then take care of othe 30X codes that might pop up at new destination.
This works for a request:
GET: https://myproxy.net/basepath?redirect=https://destination.com/somepath/uuid
What do I need to do to make it work for:
GET: https://myproxy.net/basepath?redirect=https%3A%2F%2Fdestination.com%2Fsomepath%2Fuuid
Additionally as part of spec it has to be pure nginx, not additional modules, lua etc.
Thanks!
Actually, proxy_pass does normalisation by default, but it only affects $uri part. Thus you only need to decode the beginning of the passed string to get it working:
location / {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
if ( $arg_redirect ~ (.+)%3A%2F%2F(.+) ){ # fix :// between scheme and destination
set $arg_redirect $1://$2;
}
if ( $arg_redirect ~ (.+?)%3A(.*) ){ # fix : between destination and port
set $arg_redirect $1:$2;
}
if ( $arg_redirect ~ (.+?)%2F(.*) ){ # fix / after port, the rest will be decoded by proxy_pass
set $arg_redirect $1/$2;
}
proxy_pass $arg_redirect;
}
With the above I managed to access http://localhost/?redirect=http%3A%2F%2F127.0.0.1%3A81%2Fsfoo%20something%2Fs
The solution seems dirty and the only alternative using default modules is map (even less cleaner in my opinion). I'd rather split redirect argument into pieces: scheme (http or https), destination, port, and uri. With that you would be able to construct full address without rewriting:
proxy_pass $arg_scheme://$arg_dest:$arg_port/$arg_uri
Ok, there is very weird and curious solution
server {
listen 80;
resolver x.x.x.x;
location /basepath {
if ($arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass http://127.0.0.1:80/basepath/$arg_redirect;
}
location ~ ^/basepath/(?<proto>\w+):/(?<redir>.+)$ {
proxy_pass $proto://$redir;
}
}
Nginx does not encode path with variables in proxy_pass and send it as is. So, I make $arg_* part of proxy_pass uri, send request to self and nginx will receive new request which will be decoded.
But because Nginx will clean path and replace // to / I split protocol part in regexp.
And ... I would never recommend using this solution, but it works :)
try like this and let me know if it works
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
set_unescape_uri $decodedredirect $arg_redirect;
proxy_pass $decodedredirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}

Calling external api in Nginx location section

I am trying to resolve proxy_pass value dynamically (through web api) in nginx.
I need something like below;
Example taken from: https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/
location /proxy-pass-uri {
set $urlToProxy = CallWebAPI("http://localhost:8081/resolver?url=" + $url);
proxy_pass $urlToProxy;
}
So, my question is that, is it possible to make HTTP request or to write method such as CallWebAPI?
I know it might be a bad practice, but the website I am dealing with has thousands of web urls, which are mapped as key-value pairs, and 90% of them does not obey any specific regex rules. So I have content mapped database, and I need to fetch incoming url with content dynamically.
I am trying to use a very light web service to look up URLs from redis, and return proxy url.
Would this be a valid scenario, or is there any other built in solution in nginx like this?
I doubt this can be done with "pure" nginx, but this definitely can be done with openresty or ngx_http_lua_module with the help of ngx.location.capture method. For example:
resolver 8.8.8.8;
location ~/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("http://localhost:8081/resolver?url=" .. ngx.var.url)
ngx.var.proxy = res.body
}
proxy_pass $proxy$url;
}
There is also an ngx_http_js_module (documentation, GitHub) which have an ability to do subrequests (example), but I never used it and cannot tell if it can be used this way.
Important update
After almost a three years since this answer was written, it comes that I needed the similar functionality myself, and it turns out that the above answer is completely broken and unworkable. You can't do a subrequest via ngx.location.capture to anything else but to some other nginx location. So the correct (checked and confirmed to be workable) example for the above question is
resolver 8.8.8.8;
location /resolver {
internal;
proxy_pass http://localhost:8081;
}
location ~ ^/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("/resolver?url=" .. ngx.var.url)
if res.status == ngx.HTTP_OK then
ngx.var.proxy = res.body
else
ngx.exit(res.status)
end
}
proxy_pass $proxy$url$is_args$args;
}
The above example assumes that the proxy resolution service is really expecting request in a /resolver?url=<uri> form. The location /resolver { ... } while being internal behaves like any other prefix location, so if the /resolver prefix for that location cannot be used for some reason, the same can be written as
resolver 8.8.8.8;
location /get_proxy {
internal;
proxy_pass http://localhost:8081/resolver;
}
location ~ ^/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("/get_proxy?url=" .. ngx.var.url)
if res.status == ngx.HTTP_OK then
ngx.var.proxy = res.body
else
ngx.exit(res.status)
end
}
proxy_pass $proxy$url$is_args$args;
}

How to disable logging images in nginx but still allow the get request?

I'm trying to log only java-script files request in the nginx access_log.
I tried using the following code i found on this site:
location ~* ^.+.(jpg|jpeg|gif|css|png|html|htm|ico|xml|svg)$ {
access_log off;
}
the problem is it doesn't allow the get request at all and i get a 404 error when trying to run the html file that executes the js file in the browse.
I want everything to work just the same but for the access log to log only request for js files.
How do i do that?
Put it in the server block and make sure that the "root" is correctly set up. It does work
Working example:
location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
expires +60d;
access_log off;
}
I have this in the server block and not a location block.
Alternatively you can keep all requests within single location but use access_log with condidional if operator to disable images logging:
map $request_uri $is_loggable {
~* ^.+\.(jpg|jpeg|gif|css|png|html|htm|ico|xml|svg)$ 0;
default 1;
}
server {
location / {
access_log /path/to/log/file combined if=$is_loggable;
...
}
}
Here combined is a name of default log format.
You say that you want to log only java-script files, so actually you can use even simplier solution:
map $request_uri $is_loggable {
~* ^.+\.js$ 1;
default 0;
}

nginx limit_req_zone doesn't seem to work

I am trying to setup a global limit to a certain uri using nginx via openresty
with the following config, if I curl this box I get a 204 no matter how many times I request it per minute.
worker_processes 1;
error_log logs/error.log;
events {
}
http {
log_format spec_format '$request_uri $const $status';
access_log logs/access.log spec_format;#off;
resolver 10.0.0.2;
limit_req_log_level error;
limit_req_zone $const zone=one:100k rate=1r/m;
server {
set $const 1;
listen 80;
location / {
return 200 "invalid url";
}
location ~* /request/? {
limit_req zone=one burst=1 nodelay;
return 204;
}
location /health/ {
return 200 "healthy";
}
}
}
From the docs I can't find anything obvious (Ive tried switching things around a lot).
In case it helps, the box is running on AWS behind an EIP and Ubuntu 13.10. I'm using openresty-1.5.8.1 from openresty.org.
Also, the actual limit I want to work is for around 24000r/s and there are other settings I thought might be conflicting, but even stripped down it doesn't behave like I thought it should.
The problem may be that $const is not yet set when limit_req_zone processes it. Are you seeing the correct $cost value in the log?
The limit_req_module will ignore empty values, from the docs:
The key is any non-empty value of the specified variable (empty values
are not accounted).

NGINX too long location (massive?)

NGINX
Have a locations like:
location ~ "^\/(place1|place2|...|place50)\/(service1|service2|...|service80)\-(else1|else2|...|else90)\/"
{...}
location ~ "^\/(word1|word2|...|word70)\/(place1|place2|...|place50)\-(else1|else2|...|else90)\/"
{...}
location ~ "^\/..."
The problem is there are a lot of places & services & words & else. So locations are awfully long strings. Is it possible to make them shorter? Maybe massive of places and massive of services and so on? Or something? Who has some experience?
examples of URI i want to match
/place23/service17-else87/
or
/world33/place42-else15/
and any combinations
For each location we are going to use set of rules. In order to use cache and unload our Apache
#proxy_cache start
set $do_not_cache 0;
# POST requests and urls with a query string should always go to PHP
if ($request_method = POST) {
set $do_not_cache 1;
}
if ($query_string != "") {
set $do_not_cache 1;
}
# Don't use the cache for logged in users or REBent commenters
if ($http_cookie ~* "wordpress_logged_in|bn_my_logged") {
set $do_not_cache 1;
}
if ($args ~* (show) ) {
set $do_not_cache 1;
}
ssi_types "*";
ssi on;
if ($do_not_cache = 0) {
set $memcached_key "SMREG|$request_uri";
memcached_pass memc_server;
ssi on;
}
You can use regular expressions in location blocks to make them match multiple things, rather than listing them all by hand.
e.g.
location ~* ^/world(\d{1,2})/place(\d{1,2})-else(\d{1,2})/ {
set $originalURI $uri;
fastcgi_param QUERY_STRING q=$originalURI&world=$1&place=$2&else=$3;
# or however you're passing it to your web server.
}
However as Mohammad AbuShady implied, what you're doing seems quite dumb.
You should be letting your application do the routing, and deciding whether things can be cached or not, and then using something that is actually designed to cache web pages e.g. Varnish, rather than trying to force application logic into Nginx.

Resources