I am trying to setup a global limit to a certain uri using nginx via openresty
with the following config, if I curl this box I get a 204 no matter how many times I request it per minute.
worker_processes 1;
error_log logs/error.log;
events {
}
http {
log_format spec_format '$request_uri $const $status';
access_log logs/access.log spec_format;#off;
resolver 10.0.0.2;
limit_req_log_level error;
limit_req_zone $const zone=one:100k rate=1r/m;
server {
set $const 1;
listen 80;
location / {
return 200 "invalid url";
}
location ~* /request/? {
limit_req zone=one burst=1 nodelay;
return 204;
}
location /health/ {
return 200 "healthy";
}
}
}
From the docs I can't find anything obvious (Ive tried switching things around a lot).
In case it helps, the box is running on AWS behind an EIP and Ubuntu 13.10. I'm using openresty-1.5.8.1 from openresty.org.
Also, the actual limit I want to work is for around 24000r/s and there are other settings I thought might be conflicting, but even stripped down it doesn't behave like I thought it should.
The problem may be that $const is not yet set when limit_req_zone processes it. Are you seeing the correct $cost value in the log?
The limit_req_module will ignore empty values, from the docs:
The key is any non-empty value of the specified variable (empty values
are not accounted).
Related
I want to impose a request limit for uncached content on my NGINX reverse proxy. I have multiple locations defined and content can get cached or won't get cached due to other rules. So I can not set a request limit just for a location, I have to handle this differently.
According to the documentation in https://www.nginx.com/blog/rate-limiting-nginx/#Advanced-Configuration-Examples, I can use the map feature in order to impose a request limit. So I tried this and created following configuration snippet:
map $upstream_cache_status $limit {
default 1;
MISS 1;
HIT 0;
}
map $limit $limit_key {
0 "";
1 $binary_remote_addr;
}
limit_req_zone $limit_key zone=req_zone:10m rate=5r/s;
So in order to test my map first, I have added following to my location:
add_header X-Test $limit;
And I see that it works! Every resource that is cached ($upstream_cache_status = HIT), $limit seems to be 0. Every uncached content ($upstream_cache_status = MISS), $limit is 1.
Now comes the weird behaviour. As soon as I add limit_req zone=req_zone burst=10 nodelay; into my location, $limit seems to be stuck at 1, no matter if the $upstream_cache_status is HIT or MISS.
The location looks like this:
location ~* \.(jpg|jpeg|png|gif|webp|svg|svgz|ico|pdf|doc|docx|xls|xlsx|csv|zip|gz|woff|woff2|ttf|otf|eot)$ {
limit_req zone=req_zone burst=10 nodelay;
[...]
add_header X-Test $limit;
[...]
}
Is this a NGINX bug or am I missing something here? NGINX version is 1.20.1 on AlmaLinux 8.5.
Rate limiting works first, on request phase.
Caching works later, on content phase (guess).
So, when limiter works, there is no information about cache status yet.
I had following nginx configuration:
worker_processes 1;
error_log logs/error.log;
events {
worker_connections 1024;
}
http {
limit_req_zone $request_uri zone=by_uri_6000:10m rate=6000r/s;
server {
listen 80;
server_name "openresty.com";
location = /limit-6000 {
limit_req zone=by_uri_6000 nodelay;
return 200 "ok"
}
}
}
But it is not working at all when testing using wrk:
wrk -c100 -t10 -d5s http://localhost/limit-6000
Running 5s test # http://localhost/
10 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.57ms 1.65ms 23.47ms 81.17%
Req/Sec 1.81k 495.91 9.27k 99.60%
90400 requests in 5.10s, 17.93MB read
Requests/sec: 17713.29
Transfer/sec: 3.51MB
But, if I change my configuration to something like below (I was using openresty in fact):
location = /limit-6000 {
limit_req zone=by_uri_6000 nodelay;
default_type 'application/json';
content_by_lua_block {
ngx.say('{"code": 0, "msg": "成功"}')
}
it works then, what could be the reason, I didn't see any explanation in official document.
nginx return directive works on rewrite phase, it immediately returns the result.
I believe ngx_http_limit_req_module works on access phase. So with return directive ngx_http_limit_req_module doesn't have any chances to be in game.
I'm actually a little surprised that I couldn't find anything after a couple hours of googling, but the problem is as follows:
I want nginx to serve as my throttle for my API.
My config file contains a well-cited example of limit_req_zone:
limit_req_zone $binary_remote_addr zone=limit:2m rate=10r/m;
along with my location directive containing the expected limit_req zone=limit nodelay;
I would love to have nginx attach headers to the response message for both the X-RateLimit-Remaining and X-RateLimit-Reset attributes. Basically have nginx use the active count of the rate=10r/m to populate X-RateLimit-Remaining and timeframe of the same rate=10r/m value to populate X-RateLimit-Reset with how many seconds are left before a refresh.
http {
limit_req_zone $binary_remote_addr zone=login:10m rate=2r/s;
limit_req_status 429;
limit_conn_status 429;
server {
listen 80;
server_name [removed];
location / {
limit_req zone=limit nodelay;
proxy_pass http://reverse-proxy-example;
add_header X-RateLimit-Remaining [nginx variable?];
add_header X-RateLimit-Reset [nginx variable?]
}
}
Thoughts? Possible? Would love to avoid hitting the application to get these numbers.
I would say that this isn't possible with the upstream version of nginx.
You can find the documentation for the limit_req directive through http://nginx.org/r/limit_req, which redirects to http://nginx.org/docs/http/ngx_http_limit_req_module.html#limit_req, which conclusively shows that the module doesn't have any known variables within it.
Looking at http://ngx.su/src/http/modules/ngx_http_limit_req_module.c confirms the conjecture.
Another option is to look at http://nginx.org/docs/varindex.html, which lists all the variables — looking for limit will only get you to $limit_rate, which is an unrelated variable.
P.S. Consider that the limit_req is done through a leaky bucket method.
https://en.wikipedia.org/wiki/Leaky_bucket
Without going into further details or making stuff up (the wikipedia article is huge!), I'd guess that it may not be entirely trivial to present this information to the end user in a consistent and actionable manner.
Just like #cnst's answer, there is no an approprivate varible to save the value you want, if you really want the message, you can implement a lua function to save this message.
The nginx config would like this:
http {
limit_req_zone $binary_remote_addr zone=login:10m rate=2r/s;
limit_req_status 429;
limit_conn_status 429;
server {
listen 80;
server_name [removed];
set $x-ratelimit-ramaining 0;
set $x-ratelimit-reset 0;
access_by_lua_file your_file_name.lua;
location / {
limit_req zone=limit nodelay;
proxy_pass http://reverse-proxy-example;
add_header X-RateLimit-Remaining $x-rate-limit-remaining;
add_header X-RateLimit-Reset $x-ratelimit-reset;
}
}
thus, each request will trigger the lua script, and you can update the variables $x-rate-limit-remaining $x-ratelimit-reset in your lua function.
Packngo from Packet has a function that might be a good fit for what you need to do.
Sample from packngo:
func (r *Response) populateRate() {
// parse the rate limit headers and populate Response.Rate
if limit := r.Header.Get(headerRateLimit); limit != "" {
r.Rate.RequestLimit, _ = strconv.Atoi(limit)
}
if remaining := r.Header.Get(headerRateRemaining); remaining != "" {
r.Rate.RequestsRemaining, _ = strconv.Atoi(remaining)
}
if reset := r.Header.Get(headerRateReset); reset != "" {
if v, _ := strconv.ParseInt(reset, 10, 64); v != 0 {
r.Rate.Reset = Timestamp{time.Unix(v, 0)}
}
}
}
I'm trying to log only java-script files request in the nginx access_log.
I tried using the following code i found on this site:
location ~* ^.+.(jpg|jpeg|gif|css|png|html|htm|ico|xml|svg)$ {
access_log off;
}
the problem is it doesn't allow the get request at all and i get a 404 error when trying to run the html file that executes the js file in the browse.
I want everything to work just the same but for the access log to log only request for js files.
How do i do that?
Put it in the server block and make sure that the "root" is correctly set up. It does work
Working example:
location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
expires +60d;
access_log off;
}
I have this in the server block and not a location block.
Alternatively you can keep all requests within single location but use access_log with condidional if operator to disable images logging:
map $request_uri $is_loggable {
~* ^.+\.(jpg|jpeg|gif|css|png|html|htm|ico|xml|svg)$ 0;
default 1;
}
server {
location / {
access_log /path/to/log/file combined if=$is_loggable;
...
}
}
Here combined is a name of default log format.
You say that you want to log only java-script files, so actually you can use even simplier solution:
map $request_uri $is_loggable {
~* ^.+\.js$ 1;
default 0;
}
I'm trying to figure out how to do the following:
Request is coming in.
HttpLuaModule performs some action against the request. If request is valid than Lua will finish processing with ngx.exit(202). But there are some conditions that may (and will) occur during the processing and nginx might return 403 , 404, 503 Errors.
What I want to do is to write to access logs only requests that have 200 Status code.
Basically I would like to do something like this:
location /foo {
content_by_lua_file "/opt/nginx/lua/process.lua";
if (status == 200) {
access_log "/path/to/the/access_log"
}
I'm very new to both nginx and lua so for me it's a bit of a challenge to figure out where to place and if statement (ether after content_by_lua_file or in side lua file) and what this if statement should look like.
nginx 1.7.0+ allows using an if condition in access_log directive itself.
access_log path [format [buffer=size [flush=time]] [if=condition]];
The if parameter (1.7.0) enables conditional logging.
A request will not be logged if the condition evaluates to “0” or an empty string
Combined with map directive its possible to send log events to different logs based on various conditions.
http {
map $status $normal {
~^2 1;
default 0;
}
map $status $abnormal {
~^2 0;
default 1;
}
map $remote_addr $islocal {
~^127 1;
default 0;
}
server {
access_log logs/access.log combined if=$normal;
access_log logs/access_abnormal.log combined if=$abnormal;
access_log logs/access_local.log combined if=$islocal;
}
}
http://nginx.org/en/docs/http/ngx_http_log_module.html
http://nginx.org/en/docs/http/ngx_http_map_module.html
you can do it by using ngx.log and log_by_lua directives.
location /conditional_log{
log_by_lua 'if ngx.status == 200 then ngx.log(ngx.ERR, "It is 200") end';
content_by_lua 'ngx.say("I am ok") ngx.exit(200)';
}
In the above code, we use log_by_lua which is called while running in log phase. In that if ngx.status == 200, we use ngx.log to trigger the logging using ngx.log.
This will write to error_log. Not sure how to write it to access_log.
For reference
http://wiki.nginx.org/HttpLuaModule#ngx.log
http://wiki.nginx.org/HttpLuaModule#log_by_lua
In every question is a part of answer. You were very close:
if ($status != "200") {
access_log off;
}
Check info for version availability here.
http://nginx.org/en/docs/http/ngx_http_core_module.html#variables
Also, almost all access log format vars are available in "modern" versions:
http://nginx.org/en/docs/http/ngx_http_log_module.html
This is the solution I came up with:
auth.lua
-- Some logic goes here
-- ....
-- ....
ngx.var.return_status = 200
nginx.conf
http {
lua_package_path .....;
lua_package_cpath ....;
rewrite_by_lua_no_postpone on;
server {
set $return_status 1;
location /foo {
rewrite_by_lua_file "<apth_to_aut.lua";
if ($return_status = 200) {
access_log <path_to_access_log> format;
return 200;
}
}
}
}