Using https://openresty.org/en/ I have the following nginx.conf file
I'm trying to put in a newline \n between each request header in the set_by_lua section which I then output to the access.log file.
http {
# Trying to see the request headers nicely in the access.log
# can't seem to get a nice line feed in from lua
log_format main '(ra:)$remote_addr - (ru:)$remote_user (time_local:)[$time_local] (request:)"$request" (status:)$status (bodybytessent:)$body_bytes_sent (http_referer)"$http_referer" (useragent:)"$http_user_agent" (xforward:)"$http_x_forwarded_for" \n(requestheaders:)$request_headers';
access_log logs/access.log main;
server {
listen 80;
server_name localhost;
location / {
set_by_lua $request_headers '
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
# Here is the problem - the \n does not work
# just want to append a line feed to this string
request_headers_all = request_headers_all .. ""..k..": "..v..";" .. "\n"
end
return request_headers_all
';
root html;
index index.html index.htm;
}
...continued
The error is:
2020/06/29 11:36:04 [error] 8204#8204: *25 failed to load inlined Lua code: set_by_lua:6: unfinished string near '"'
This line is the problem:
request_headers_all = request_headers_all .. ""..k..": "..v..";" .. "\n"
So it is some sort of escaping issue. Lua ignore Escape Sequence which I'm lost with.
Base OS is Ubuntu 18.04. and I'm trying to get something like this into the access.log file:
instead of this:
(ra:)37.152.225.237 - (ru:)- (time_local:)[29/Jun/2020:11:47:46 +0000] (request:)"GET / HTTP/1.1" (status:)200 (bodybytessent:)422 (http_referer)"-" (useragent:)"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0" (xforward:)"-"
(requestheaders:)host: openrestytest836.westeurope.cloudapp.azure.com;accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8;upgrade-insecure-requests: 1;cache-control: no-cache;accept-language: en-GB,en;q=0.5;user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0;pragma: no-cache;connection: keep-alive;accept-encoding: gzip, deflate;
There are a lot of ways of doing this, but I'm curious as to how to get the newline characters into the Lua generated string.
Thanks to #un.def in the comments above who fixed the Lua issue which is now:
set_by_lua_block $request_headers {
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
request_headers_all = request_headers_all .. ""..k..": "..v.."\n"
end
return request_headers_all
}
I've not tried the suggested table strategy yet.
From http://nginx.org/en/docs/http/ngx_http_log_module.html
I had to set esacpe=none in the log_format also:
log_format main escape=none '(remote_addr:)$remote_addr - (remote_user:)$remote_user (time_local:)[$time_local] (request:)"$request" (status:)$status (bodybytessent:)$body_bytes_sent (http_referer)"$http_referer" (useragent:)"$http_user_agent" (xforward:)"$http_x_forwarded_for" \n(requestheaders:)\n$request_headers';
which gives the desired output:
Related
I am using lua inside nginx, below is the code to encode a string:
set_by_lua $base64_credential '
set $es_username os.getenv("ES_USERNAME");
set $es_pwd os.getenv("ES_PWD");
return ngx.encode_base64(ngx.var.es_username+":"+ngx.var.es_pwd)
'
after launching the server I got below error:
2021/11/18 01:58:01 [error] 7#7: *151 failed to load inlined Lua code: set_by_lua:2: '=' expected near '$', client: 10.0.6.61, server: localhost, request: "GET /health HTTP/1.1", host: "10.0.1.246:8080"
I use the syntax from this doc https://github.com/openresty/lua-nginx-module#set_by_lua and it doesn't use = sign when set a variable. What did I do wrong?
Again, you made a couple of errors. Lua operator for string concatenation is ... Lua doesn't expect semicolons between the operators. You have a weird mix of lua and nginx configuration syntax. If you don't need those $es_username and $es_pwd variables elsewhere, use
set_by_lua $base64_credential '
local es_username = os.getenv("ES_USERNAME")
local es_pwd = os.getenv("ES_PWD")
return ngx.encode_base64(es_username .. ":" .. es_pwd)
';
If you need those variables elsewhere, then your way is
set_by_lua $es_username 'return os.getenv("ES_USERNAME")';
set_by_lua $es_pwd 'return os.getenv("ES_PWD")';
set_by_lua $base64_credential 'return ngx.encode_base64(ngx.var.es_username .. ":" .. ngx.var.es_pwd)';
I don't use Lua, but need to use it with Openresty (nginx) as provided in the link.
Openresty has a lua module which I managed to install and run the Openresty nginx version correctly, the website working.
This answer shows how to concatenate headers into a string $request_headers:
set_by_lua $request_headers '
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
end
return request_headers_all
';
I changed the format from ""..k..": "..v..";" to "["..k..": "..v.."]" in the lua function above.
Log format:
log_format log_realip 'Host: "$host", Via : "$http_via", RemoteAddr: "$remote_addr", XForwardedFor: "$h
ttp_x_forwarded_for", 'RealIPRemoteAddr: "$realip_remote_addr" - $status - "$request_uri" - **"[HEADERS]" - $request_headers';**
Host: "192.168.1.4", Via : "-",
//trimmed just to show the [HEADERS]
....
"[HEADERS]" - [sec-ch-ua: \x22Chromium\x22;v=\x2288\x22, \x22Google Chrome\x22;v=\x228
8\x22, \x22;Not A Brand\x22;v=\x2299\x22][sec-ch-ua-mobile: ?0][cookie: __utmz=abcdef; frontend=abcdef; adminhtml=abcdef
08; TestName=Some Value][upgrade-insecure-requests: 1][accept-language: en-US,en;q=0.9][user-agent: Mozilla/5.0
(Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36][accept
-encoding: gzip, deflate, br][accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/we
bp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9][sec-fetch-dest: document][host: 192.168.1.4][se
c-fetch-user: ?1][connection: keep-alive][sec-fetch-mode: navigate][cache-control: max-age=0][sec-fetch-site: n
one
When using log_format with $request_headers string I get all the headers in one line, but I am trying to create a newline \n to break the string into lines. The example above is where I added \n but doesn't seem to output break to the log file.
I understand the request_headers_all .. concatenates the string, but what is happening here with the key k and value v : ""..k..": "..v..\n""?
What is the "..variablename.." doing, is this how variables are always used inside Lua strings?
How would I be able to create a line break in that string? Or is it possible that nginx(openresty) doesn't output the newline?
you add the \n to a wrong place, you can change to
request_headers_all = request_headers_all .. "["..k..": "..v.."]\n" for a newline log.
In lua, the .. is a concat operator, to concat to strings, for example:
print("hello" .. "world")
get the result helloworld.
your code \n"]" have syntax error, because \n not in a string.
lua strings can not directly use variables, usually, lua use string.format for complex string. for example:
local test = "hello"
string.format("%s world",test) -- hello world
you can use string.format for you string concat.
also, you can use table.concat to concat strings.
for example:
local test = {}
table.insert(test, "hello")
table.insert(test, "world")
local concated_string = table.concat(test, ' ')
print(concated_string) -- hello world
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
contains a syntax error. replace "["..k..": "..v..\n"]" with "["..k..": "..v.."]\n"
Newline needs to be inside the quotes as it is part of the string and it will probably make sense to add the new line after the bracket.
What is the "..variablename.." doing, is this how variables are always
used inside Lua strings?
using the concatenation operator on a variable concatenates a string value, concatenates its string representation if it is a number or invokes __concat or raises an error if neither of those is true.
Read https://www.lua.org/manual/5.4/manual.html#3.4.6
The above answers gave me some guidance, but the formats suggested still didn't work. After playing around with string.format("%s %s\n", k, v), string.format("%s %s\\n", k, v) I still got unfinished string errors or no newline output. (Tried to escape string in second example).
Based on the answers given I assumed the answers gave correct lua information, so decided most likely lua + openresty does something different.
I will attempt to update the title to reflect more specific requirements
TLDR
Openresty + lua string manipulation with special characters might not work as expected, even when using string.format()
Change from set_by_lua which returns a string to set_by_lua_block which allows string.format() or string concatenation better.
Update nginx configuration with your custom/existing log_format and add the switch escape=none.
Full Explanation
Investigating the provided link answer set_by_lua function documentation :
NOTE Use of this directive is discouraged following the v0.9.17 release. Use the set_by_lua_block directive instead.
So from the original set_by_lua from the link:
set_by_lua $request_headers '
return 'stringvalue'
';
I changed to set_by_lua_block function:
this directive inlines the Lua source directly inside a pair of curly braces ({}) instead of in an Nginx string literal (which requires special character escaping)
set_by_lua_block $request_headers{
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
local rowtext = ""
rowtext = string.format("[%s %s]\n", k, v)
request_headers_all = request_headers_all .. rowtext
end
return request_headers_all
}
The important part is that this _block {} function escapes the special characters correctly.
After that I received output in the log files as : x0A (newline character literal).
The final step then is to update the nginx.conf file with the custom log_format and add escape=none:
log_format log_realip escape=none "$nginx_variables"
The main question/problem here is how does the wrapper software handle newline characters/escapes, is it expecting "\n"? or is it expecting "\r\n"?
Ultimately the new line does not actually exist until it is interpreted and printed and you are creating one massive string that gets returned from the Lua engine to the wrapper, so it is up to the wrapper software on how to interpret new lines.
Edit: I missed that this was already answered by using the other parsing function.
Additionally the docs state to use the original parsing function escapes need to be double escaped.
Here, \\\\d+ is stripped down to \\d+ by the Nginx config file parser and this is further stripped down to \d+ by the Lua language parser before running.
I'm trying to get metrics from nginx about the total connections when using the nginx stream module. I know there is stub_status option but it looks like its only for http and not stream.
I found the prometheus lua plugin which has stream, but I was not able to figure out the right conf for it. I only get the built-in error metric but not the metric defined. Here's the conf and the response. Whats also weird is that the response is in the header and not body.
stream {
lua_shared_dict prometheus_metrics 10M;
lua_package_path "/tmp/nginx-lua-prometheus/?.lua;/usr/local/openresty/luajit/lib/lua/5.1/?.lua;;";
init_by_lua '
prometheus = require("prometheus").init("prometheus_metrics")
metric_requests = prometheus:counter("nginx_http_requests_total", "Number of HTTP requests", {"test"})
metric_connections = prometheus:gauge("nginx_http_connections", "Number of HTTP connections", {"test"})';
log_by_lua_block {
metric_requests:inc(1, {"test"})
}
server {
listen 9145;
content_by_lua '
local sock = assert(ngx.req.socket(true))
local data = sock:receive()
local location = "GET /metrics"
if string.sub(data, 1, string.len(location)) == location then
ngx.say("HTTP/1.1 200 OK")
ngx.say("Content-Type: text/plain")
ngx.say("yo")
ngx.say(table.concat(prometheus:metric_data(), ""))
else
ngx.say("HTTP/1.1 404 Not Found")
end
';
}
}
curl localhost:9145/metrics -v
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9145 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:9145
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain
< # HELP nginx_metric_errors_total Number of nginx-lua-prometheus errors
< # TYPE nginx_metric_errors_total counter
< nginx_metric_errors_total 0
* no chunk, no close, no size. Assume close to signal end
<
* Curl_http_done: called premature == 0
* Closing connection 0
https://github.com/knyar/nginx-lua-prometheus/issues/77
Answered here
log_by_lua_block {
metric_connections:set(ngx.var.connections_active, {"active"})
metric_connections:set(ngx.var.connections_reading, {"reading"})
metric_connections:set(ngx.var.connections_writing, {"writing"})
metric_connections:set(ngx.var.connections_waiting, {"waiting"})
metric_requests:inc(1, {"test"})
}
I'm trying to create a fail2ban filter for wordpress under nginx, but its not working and need some help.
The Log:
111.111.111.111 - - [27/Oct/2019:02:54:48 +0200] "POST /wp-login.php HTTP/1.1" 200 1697 "http://my-wordpress.com/wp-login.php" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.1 Mobile/15E148 Safari/604.1"
Filter:
[INCLUDES]
before = common.conf
[Definition]
failregexe = <HOST> - - .* "POST /wp-login.php .* 200
ignoreregex =
Jail:
[wordpress]
enabled = true
port = http,https
filter = wordpress
logpath = /var/log/nginx/access.log
maxretry = 3
bantime = 3600
Test Regex
#fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/wordpress.conf
Running tests
=============
Use failregex filter file : wordpress, basedir: /etc/fail2ban
Use datepattern : Default Detectors
Use log file : /var/log/nginx/access.log
Use encoding : UTF-8
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Date template hits:
Lines: 72 lines, 0 ignored, 0 matched, 72 missed
[processed in 0.01 sec]
Missed line(s): too many to print. Use --print-all-missed to print all 72 lines
Whats wrong on this regex?
I tried also
^<HOST>.*"POST./wp-login.php.*200.*$
^<HOST> - - .* "POST /wp-login.php .* 200
and many more....
There is a typo (remove last e):
- failregexe = ...
+ failregex =
BTW: do you really want to ban 200th response? (200 means OK).
Additionally your regex is too "vulnerable", better use something anchored and more precise, like this:
^<HOST> - \S+ \[\] "[A-Z]+ /wp-login.php [^"]+" 200
Or even use (?!- )\S+ ... [54]0\d+ instead of \S+ ... 200 if you want capture other codes and for users with real username only (so would ignore handshake requests where user logged as -).
I am using openresty with lua. When I was loading torch without cutorch,cunn(GPU) it was working fine. But it is unable to load cutorch module.
Here is my nginx.conf
error_log stderr notice;
daemon off;
events{}
http {
lua_code_cache on;
#lua_package_cpath '/usr/local/cuda/include/?.so;;';
init_by_lua_file 'evaluate.lua';
server {
listen 6788;
lua_code_cache on;
lua_need_request_body on;
client_body_buffer_size 10M;
client_max_body_size 10M;
location /ff {
# only permit POST requests
if ($request_method !~ ^(POST)$ ) {
return 403;
}
content_by_lua '
local body = ngx.req.get_body_data()
if body == nil then
ngx.say("empty body")
else
local resp =FeedForward(body)
ngx.say(cjson.encode({result=resp}))
end
';
}
}
}
Here is my evaluate.lua code
-- load the required lua models
torch = require("torch")
nn = require("nn")
gm = require("image")
cutorch = require "cutorch"
cunn = require "cunn"
cjson = require("cjson")
-- model
modelPath='./model.t7'
ninputs = 3
model = nn.Sequential()
model:add(nn.Linear(ninputs,2))
-- let’s save a dummy model
-- to demonstrate the functionality
torch.save(modelPath, model)
-- load a pretrained model
net = torch.load(modelPath)
net:cuda()
net:evaluate()
-- our end point function
-- this function is called by the ngx server
-- accepts a json body
function FeedForward(json)
print("starting")
-- decode and extract field x
local data = cjson.decode(json)
local input = torch.Tensor(data.x)
local response = {}
-- example checks
if input == nil then
print("No input given")
elseif input:size(1) ~=ninputs then
print("Wrong input size")
else
-- evaluat the input and create a response
local output = net:forward(input:cuda()):float()
-- from tensor to table
for i=1,output:size(1) do
response[i] = output[i]
end
end
-- return response
return response
end
I am trying to run the model using following commands
/usr/local/openresty/nginx/sbin/nginx -p "$(pwd)" -c "nginx.conf"
It's starting up fine but when I am sending curl request like
curl -H "Content-Type: application/json" -X POST -d '{"x":[1,2,3]}' http://localhost:6788/ff
I am getting following error.
2016/09/29 12:59:59 [notice] 10355#0: *1 [lua] evaluate.lua:28: FeedForward(): starting, client: 127.0.0.1, server: , request: "POST /ff HTTP/1.1", host: "localhost:6788"
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-700/cutorch/lib/THC/generic/THCStorage.cu line=40 error=3 : initialization error
2016/09/29 12:59:59 [error] 10355#0: *1 lua entry thread aborted: runtime error: unknown reason
stack traceback:
coroutine 0:
[C]: in function 'resize'
/home/ubuntu/torch/install/share/lua/5.1/cutorch/Tensor.lua:14: in function 'cuda'
/rootpath/evaluate.lua:41: in function 'FeedForward'
content_by_lua(nginx.conf:31):7: in function <content_by_lua(nginx.conf:31):1>, client: 127.0.0.1, server: , request: "POST /ff HTTP/1.1", host: "localhost:6788"
Without cutorch the model is running fine like If I remove
net:cuda()
and replace line
local output = net:forward(input:cuda()):float()
by
local output = net:forward(input):float()
It's working fine. Also I tried to run evaluate.lua using th and it's working fine there with cutorch and cunn packages.