Nginx + LUA, how to output file? - nginx

Having trouble with file output in Nginx + Lua. I chosen LUA, because nginx logic is pretty complicated, based on referrer or subdomains, etc.
Having request like /img/am1/s/1.jpg I need to check if file exists in /somepath/am1/1.jpg. If it exists, then output it, otherwise proxy request to backend.

Ok, found it
content_by_lua '
local file = "/path..."
local f = io.open(file, "rb")
local content = f:read("*all")
f:close()
ngx.print(content)
';

If someone need to know how to output last n lines from file:
location /service-man/log {
default_type 'text/plain';
content_by_lua '
local log_path = "/path/to/log.log"
-- Opens a file in read
file = io.open(log_path, "r")
if file==nil
then
ngx.say(log_path .. " can\'t read or does not exists")
return
end
-- sets the default input file
io.input(file)
local lines = {}
-- read the lines in table lines
for line in io.lines() do
table.insert(lines, line)
end
io.close(file)
log_limit = 10
if #lines < log_limit then
log_start = 0
else
log_len = #lines
log_start = log_len - log_limit
end
local one_line = ""
for i, line in ipairs(lines) do
if i > log_start then
one_line = one_line .. line .. "\\n"
end
end
ngx.say(one_line)
';
}
Should be compatible with nginx/1.6.2 and Lua 5.3.
Please share if you know how to make it in more optimal way.

Related

Lua - Concatenation of variables (pairs) escaping special characters - Openresty Nginx

I don't use Lua, but need to use it with Openresty (nginx) as provided in the link.
Openresty has a lua module which I managed to install and run the Openresty nginx version correctly, the website working.
This answer shows how to concatenate headers into a string $request_headers:
set_by_lua $request_headers '
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
end
return request_headers_all
';
I changed the format from ""..k..": "..v..";" to "["..k..": "..v.."]" in the lua function above.
Log format:
log_format log_realip 'Host: "$host", Via : "$http_via", RemoteAddr: "$remote_addr", XForwardedFor: "$h
ttp_x_forwarded_for", 'RealIPRemoteAddr: "$realip_remote_addr" - $status - "$request_uri" - **"[HEADERS]" - $request_headers';**
Host: "192.168.1.4", Via : "-",
//trimmed just to show the [HEADERS]
....
"[HEADERS]" - [sec-ch-ua: \x22Chromium\x22;v=\x2288\x22, \x22Google Chrome\x22;v=\x228
8\x22, \x22;Not A Brand\x22;v=\x2299\x22][sec-ch-ua-mobile: ?0][cookie: __utmz=abcdef; frontend=abcdef; adminhtml=abcdef
08; TestName=Some Value][upgrade-insecure-requests: 1][accept-language: en-US,en;q=0.9][user-agent: Mozilla/5.0
(Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36][accept
-encoding: gzip, deflate, br][accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/we
bp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9][sec-fetch-dest: document][host: 192.168.1.4][se
c-fetch-user: ?1][connection: keep-alive][sec-fetch-mode: navigate][cache-control: max-age=0][sec-fetch-site: n
one
When using log_format with $request_headers string I get all the headers in one line, but I am trying to create a newline \n to break the string into lines. The example above is where I added \n but doesn't seem to output break to the log file.
I understand the request_headers_all .. concatenates the string, but what is happening here with the key k and value v : ""..k..": "..v..\n""?
What is the "..variablename.." doing, is this how variables are always used inside Lua strings?
How would I be able to create a line break in that string? Or is it possible that nginx(openresty) doesn't output the newline?
you add the \n to a wrong place, you can change to
request_headers_all = request_headers_all .. "["..k..": "..v.."]\n" for a newline log.
In lua, the .. is a concat operator, to concat to strings, for example:
print("hello" .. "world")
get the result helloworld.
your code \n"]" have syntax error, because \n not in a string.
lua strings can not directly use variables, usually, lua use string.format for complex string. for example:
local test = "hello"
string.format("%s world",test) -- hello world
you can use string.format for you string concat.
also, you can use table.concat to concat strings.
for example:
local test = {}
table.insert(test, "hello")
table.insert(test, "world")
local concated_string = table.concat(test, ' ')
print(concated_string) -- hello world
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
contains a syntax error. replace "["..k..": "..v..\n"]" with "["..k..": "..v.."]\n"
Newline needs to be inside the quotes as it is part of the string and it will probably make sense to add the new line after the bracket.
What is the "..variablename.." doing, is this how variables are always
used inside Lua strings?
using the concatenation operator on a variable concatenates a string value, concatenates its string representation if it is a number or invokes __concat or raises an error if neither of those is true.
Read https://www.lua.org/manual/5.4/manual.html#3.4.6
The above answers gave me some guidance, but the formats suggested still didn't work. After playing around with string.format("%s %s\n", k, v), string.format("%s %s\\n", k, v) I still got unfinished string errors or no newline output. (Tried to escape string in second example).
Based on the answers given I assumed the answers gave correct lua information, so decided most likely lua + openresty does something different.
I will attempt to update the title to reflect more specific requirements
TLDR
Openresty + lua string manipulation with special characters might not work as expected, even when using string.format()
Change from set_by_lua which returns a string to set_by_lua_block which allows string.format() or string concatenation better.
Update nginx configuration with your custom/existing log_format and add the switch escape=none.
Full Explanation
Investigating the provided link answer set_by_lua function documentation :
NOTE Use of this directive is discouraged following the v0.9.17 release. Use the set_by_lua_block directive instead.
So from the original set_by_lua from the link:
set_by_lua $request_headers '
return 'stringvalue'
';
I changed to set_by_lua_block function:
this directive inlines the Lua source directly inside a pair of curly braces ({}) instead of in an Nginx string literal (which requires special character escaping)
set_by_lua_block $request_headers{
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
local rowtext = ""
rowtext = string.format("[%s %s]\n", k, v)
request_headers_all = request_headers_all .. rowtext
end
return request_headers_all
}
The important part is that this _block {} function escapes the special characters correctly.
After that I received output in the log files as : x0A (newline character literal).
The final step then is to update the nginx.conf file with the custom log_format and add escape=none:
log_format log_realip escape=none "$nginx_variables"
The main question/problem here is how does the wrapper software handle newline characters/escapes, is it expecting "\n"? or is it expecting "\r\n"?
Ultimately the new line does not actually exist until it is interpreted and printed and you are creating one massive string that gets returned from the Lua engine to the wrapper, so it is up to the wrapper software on how to interpret new lines.
Edit: I missed that this was already answered by using the other parsing function.
Additionally the docs state to use the original parsing function escapes need to be double escaped.
Here, \\\\d+ is stripped down to \\d+ by the Nginx config file parser and this is further stripped down to \d+ by the Lua language parser before running.

NGINX LUA Content-Length +1 Byte Lost

I am having an interesting bug and method issue
Lua mentions that the js_content variable has a length of 80 bytes.
But when I don't use the "Content-Length" header, firefox mentions that 81 bytes of data are transferred.
I don't know where the +1 byte excess comes from
I will be glad if you can help, an application I wrote with VBNet gives an error when I noticed that the "Content-Length" header is 80 bytes while parsing json data from my remote server, but it works fine when I add +1.
local ref_array = {1, 2, 3}
local sArray = {}
sArray["1"] = "One"
sArray["2"] = "Two"
sArray["3"] = "Tree"
local ctable = {}
for index, data in ipairs(ref_array) do
if sArray[tostring(data)] ~= nil then
local cinfo = {}
cinfo["X"] = tostring(data)
cinfo["Y"] = sArray[tostring(data)]
cinfo["Z"] = 0
table.insert(ctable, cinfo)
end
end
local js_content = cjson.encode(ctable)
ngx.header['Content-Type'] = 'application/json'
ngx.header['Content-Length'] = #js_content -- 80 byte
ngx.say(js_content)
ngx.exit(200)
I guess the problem is the Line Feed Character character at the end.
ngx.say always adds linefeed
ngx.print is just output
problem solved
Linefeed Character

LUA: parse multi-line variable and cut needed string

I have next nginx location with Lua code (used with nginx lua-resty-redis):
location = /healthcheck {
content_by_lua_block {
local red = redis:new()
used_mem_limit = 536870912
-- connect to redis
ok, err = red:connect(ngx.var.redis_host, 6379)
if not ok then
ngx.log(ngx.ERR, "Redis connection error: ", err)
ngx.status = 500
return
end
-- setting connection timeout
red:set_timeout(1000)
-- getting redis used_memory_rss
memory = red:info('memory')
red_used_memory = ...
-- set response code
if (red_used_memory >= used_mem_limit) then
http_code = 500
elseif (red_used_memory < used_mem_limit)
http_code = 200
else
http_code = 500
end
ngx.status = http_code
}
}
memory variable should be like this:
# Memory
used_memory:105157968
used_memory_human:100.29M
used_memory_rss:110387200
used_memory_rss_human:105.27M
used_memory_peak:105219456
used_memory_peak_human:100.35M
used_memory_peak_perc:99.94%
used_memory_overhead:32290608
used_memory_startup:487168
used_memory_dataset:72867360
used_memory_dataset_perc:69.62%
total_system_memory:1044770816
I need to get used_memory_rss value, so then I can compare it with used_mem_limit. So, I don't know, how to complete red_used_memory = ... line.
There seems to be only digits in possible value so use a simple regexp and then convert resulting string to number:
red_used_memory = tonumber(memory:match('used_memory_rss:(%d+)'))
Line breaks are irrelevant.

Why Lua+Nginx says it cannot call global function?

I have two simple functions that detects browser and operating system based on user agent and they are stored in file useragent.lua.
function detect_browser_platform(user_agent)
-- Here goes some string matching and similar stuff
return browser_platform
end
function detect_os_platform(user_agent)
-- Here goes some string matching and similar stuff
return os_platform
end
function detect_env_pattern(user_agent)
return detect_operating_system_platform(user_agent) .. "-" .. detect_browser_platform(user_agent) .. "-" .. ngx.var.geoip2_data_country_code
end
In virtual host configuration file, there is a line that says when request looks like /lua execute lua script: /var/www/default/test.lua.
In test.lua I have this code:
local posix = require('posix')
local redis = require('redis')
require('useragent')
-- Some code goes here
local user_agent = ngx.req.get_headers()['User-Agent']
local pattern_string = detect_env_pattern(user_agent)
ngx.say(pattern_string)
ngx.exit(200)
But for some reason when I reload nginx nginx -s reload, this codes works only first time. When I make another request it says this error:
2016/09/19 12:30:08 [error] 19201#0: *125956 lua entry thread aborted: runtime error: /var/www/default/test.lua:199: attempt to call global 'detect_env_pattern' (a nil value)
And I have no idea what is happening here. I have just started programming in Lua and don't have time to go deep with language understandings... So why am I getting this error?
Wrap it by a table:
local M={};
function detect_browser_platform(user_agent)
-- Here goes some string matching and similar stuff
return browser_platform
end
function detect_os_platform(user_agent)
-- Here goes some string matching and similar stuff
return os_platform
end
function detect_env_pattern(user_agent)
return detect_operating_system_platform(user_agent) .. "-" .. detect_browser_platform(user_agent) .. "-" .. ngx.var.geoip2_data_country_code
end
M.detect_env_pattern = detect_env_pattern
return M
in base lua file:
local useragent = require('useragent')
--.....
local user_agent = ngx.req.get_headers()['User-Agent']
local pattern_string = useragent.detect_env_pattern(user_agent)
ngx.say(pattern_string)
ngx.exit(200)

lua with Nginx : bad argument to 'getNewSessionID' (string expected, got nil)

I have compiled nginx with Lua. and i am facing a problem .
When i hit localhost i get an error in nginx error logs
2015/05/29 07:41:32 [error] 112#0: *1 lua entry thread aborted: runtime error: /opt/hello.lua:11: bad argument #1 to 'getNewSessionID' (string expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'getNewSessionID'
Nginx configurations are
location / {
default_type 'text/html';
content_by_lua_file /opt/hello.lua;
}
And my hello.lua look like
package.path = package.path ..';/opt/?.lua'
JSON = (loadfile "/opt/JSON.lua")()
local proxyFunctions = require("proxyCode")
coherenceNGIXLocation = "/coherence"
local coherenceCookie = proxyFunctions.getExistingSessionID()
-- To Create a coherance Session if not present
if (not coherenceCookie or coherenceCookie == '') then
-- ngx.log(ngx.INFO,"No Coherence Session Exists, Creating New")
local uidCookieValueEncoded = proxyFunctions.base64Encode(str,proxyFunctions.getNewSessionID())
local sessCreateResponse = proxyFunctions.coherencePut('{"sessionToken":"'..uidCookieValueEncoded..'"}')
local parsedJSON= JSON:decode(sessCreateResponse.body)
ngx.log(ngx.INFO, "Coherence Session Create")
ngx.log(ngx.INFO, "\n\n\n=============Coherence Res==============\n\n\n"..sessCreateResponse.body.."\n\n\n=============Coherence Res==============\n\n\n")
end
local client_cookie = ngx.req.get_headers()["cookie"]
-- To Create a coherance Session if not present
ngx.log(ngx.DEBUG, "\n\n\n=============REQ==============\n\n\n"..proxyFunctions.deepPrint(ngx.req.get_headers()).."\n\n\n=============REQ==============\n\n\n")
-- Set the cookie for the legacy request
ngx.req.set_header('cookie', client_cookie)
ngx.log(ngx.INFO, "\n\n\n=============REQ URI==============\n\n\n"..ngx.var.request_uri.."\n\n\n=============REQ URI==============\n\n\n")
local legacy_res = ngx.location.capture("/app"..ngx.var.request_uri, { share_all_vars = true })
ngx.log(ngx.DEBUG, "\n\n\n=============APP Res==============\n\n\n"..proxyFunctions.deepPrint(legacy_res.header).."\n\n\n=============APP Res==============\n\n\n")
if (legacy_res.status == 302) then
ngx.redirect(legacy_res.header.Location )
else
ngx.log(ngx.INFO, "Passing the Page/Layout to Maken")
local dev_res_encoded = legacy_res.body
dev_res_encoded = ngx.escape_uri(dev_res_encoded)
-- ngx.req.set_header('Cookie', ngx.req.get_headers()["cookie"])
-- ngx.req.set_header("Content-Type", "text/html") -- application/x-www-form-urlencoded -- 'page='..dev_res_encoded
ngx.req.set_header('cookie', client_cookie) -- maken/render
-- legacy_res.header('Content-Length', legacy_res.header['Content-Length']);
local maken_res = ngx.location.capture("/maken/", {method = ngx.HTTP_POST, body = legacy_res.body, share_all_vars = true} )
ngx.header['Set-Cookie'] = legacy_res.header["Set-Cookie"]
ngx.log(ngx.INFO, "Sending Back Maken Response to Client/Browser with Cookies from lagacy")
ngx.say(maken_res.body)
end
My proxyCode.lua looks like
function _M.getNewSessionID()
-- strip the sessionID length 11 cookie name before putting into the session
-- return string.sub(ngx.var.uid_set,11)
return _M.removeCookieSessionPrefixname(ngx.var.uid_set,11)
end
function _M.removeCookieSessionPrefixname(cookieName)
return string.sub(cookieName,11)
end
function _M.getIncomingAuthToken()
local cookie = ngx.req.get_headers()["Cookie"]
if (cookie ~=nil) then
local str = string.match(cookie,"AuthToken=(.*)$")
if (str ~= nil) then
return string.sub(str,11)
end
end
return nil
end
Can someone help me why i am getting this error ? i am new to Lua programming. and trying to run legacy application on my laptop.

Resources