I am having an interesting bug and method issue
Lua mentions that the js_content variable has a length of 80 bytes.
But when I don't use the "Content-Length" header, firefox mentions that 81 bytes of data are transferred.
I don't know where the +1 byte excess comes from
I will be glad if you can help, an application I wrote with VBNet gives an error when I noticed that the "Content-Length" header is 80 bytes while parsing json data from my remote server, but it works fine when I add +1.
local ref_array = {1, 2, 3}
local sArray = {}
sArray["1"] = "One"
sArray["2"] = "Two"
sArray["3"] = "Tree"
local ctable = {}
for index, data in ipairs(ref_array) do
if sArray[tostring(data)] ~= nil then
local cinfo = {}
cinfo["X"] = tostring(data)
cinfo["Y"] = sArray[tostring(data)]
cinfo["Z"] = 0
table.insert(ctable, cinfo)
end
end
local js_content = cjson.encode(ctable)
ngx.header['Content-Type'] = 'application/json'
ngx.header['Content-Length'] = #js_content -- 80 byte
ngx.say(js_content)
ngx.exit(200)
I guess the problem is the Line Feed Character character at the end.
ngx.say always adds linefeed
ngx.print is just output
problem solved
Linefeed Character
Related
I've been trying to make a request to an IPv6 address using the parseRequest function from Network.HTTP.Client (https://hackage.haskell.org/package/http-client-0.7.10/docs/Network-HTTP-Client.html) package as follows:
request <- parseRequest "http://[2001:0db8:85a3:0000:0000:8a2e:0370:7334]"
Instead of parsing it as an address/addrInfo, it is parsed as a hostname and throws the error: does not exist (Name or service not known). As a next step, I tried pointing a domain to the same IPv6 address and then using the domain name in parseRequest, then it successfully resolves that into the IPv6 address and makes the request. Is there some other way I can directly use the IPv6 address to make the request using the http-client package?
PS: I also tried without square brackets around the IP address, in this case the error is Invalid URL:
request <- parseRequest "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334"
More context:
For an IPv4 address, the getAddrInfo function generates the address as:
AddrInfo {addrFlags = [AI_NUMERICHOST], addrFamily = AF_INET, addrSocketType = Stream, addrProtocol = 6, addrAddress = 139.59.90.1:80, addrCanonName = Nothing}
whereas for IPv6 address(inside the square brackets format):
AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = 0.0.0.0:0, addrCanonName = Nothing}
and the error prints as:
(ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = 0.0.0.0:0, addrCanonName = Nothing}, host name: Just "[2001:0db8:85a3:0000:0000:8a2e:0370:7334]", service name: Just "80"): does not exist (Name or service not known))
When a literal IPv6 address is used in a URL, it should be surrounded by square brackets (as per RFC 2732) so the colons in the literal address aren't misinterpreted as some kind of port designation.
When a literal IPv6 address is resolved using the C library function getaddrinfo (or the equivalent Haskell function getAddrInfo), these functions are not required to handle these extra square brackets, and at least on Linux they don't.
Therefore, it's the responsibility of the HTTP client library to remove the square brackets from the hostname extracted from the URL before resolving the literal IPv6 address using getaddrinfo, and the http-client package doesn't do this, at least as of version 0.7.10. So, this is a bug, and I can see you've appropriately filed a bug report.
Unfortunately, I don't see an easy way to work around the issue. You can manipulate the Request after parsing to remove the square brackets from the host field, like so:
{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString (ByteString)
import qualified Data.ByteString as BS
import Network.HTTP.Client
import Network.HTTP.Types.Status (statusCode)
main :: IO ()
main = do
manager <- newManager defaultManagerSettings
request <- parseRequest "http://[::1]"
let request' = request { host = removeBrackets (host request) }
response <- httpLbs request' manager
print response
removeBrackets :: ByteString -> ByteString
removeBrackets bs =
case BS.stripPrefix "[" bs >>= BS.stripSuffix "]" of
Just bs' -> bs'
Nothing -> bs
The problem with this is that it also removes the square brackets from the value in the Host header, so the HTTP request will contain the header:
Host: ::1
instead of the correct
Host: [::1]
which may or may not cause problems, depending on the web server at the other end.
You could try using a patched http-client package. The following patch against version 0.7.10 seems to work, but I didn't test it very extensively:
diff --git a/Network/HTTP/Client/Connection.hs b/Network/HTTP/Client/Connection.hs
index 0e329cd..719822e 100644
--- a/Network/HTTP/Client/Connection.hs
+++ b/Network/HTTP/Client/Connection.hs
## -15,6 +15,7 ## module Network.HTTP.Client.Connection
import Data.ByteString (ByteString, empty)
import Data.IORef
+import Data.List (stripPrefix, isSuffixOf)
import Control.Monad
import Network.HTTP.Client.Types
import Network.Socket (Socket, HostAddress)
## -158,8 +159,12 ## withSocket :: (Socket -> IO ())
withSocket tweakSocket hostAddress' host' port' f = do
let hints = NS.defaultHints { NS.addrSocketType = NS.Stream }
addrs <- case hostAddress' of
- Nothing ->
- NS.getAddrInfo (Just hints) (Just host') (Just $ show port')
+ Nothing -> do
+ let port'' = Just $ show port'
+ case ip6Literal host' of
+ Just lit -> NS.getAddrInfo (Just hints { NS.addrFlags = [NS.AI_NUMERICHOST] })
+ (Just lit) port''
+ Nothing -> NS.getAddrInfo (Just hints) (Just host') port''
Just ha ->
return
[NS.AddrInfo
## -173,6 +178,11 ## withSocket tweakSocket hostAddress' host' port' f = do
E.bracketOnError (firstSuccessful addrs $ openSocket tweakSocket) NS.close f
+ where
+ ip6Literal h = case stripPrefix "[" h of
+ Just rest | "]" `isSuffixOf` rest -> Just (init rest)
+ _ -> Nothing
+
openSocket tweakSocket addr =
E.bracketOnError
(NS.socket (NS.addrFamily addr) (NS.addrSocketType addr)
I’m trying to rework a script I found online to control a Panasonic TV, which requires a secure/encrypted pairing to occur so I can control it remotely. (The full code here -> https://forum.logicmachine.net/showthread.php?tid=232&pid=16580#pid16580)
Because it seems to be built on LuaJIT and has some other proprietary Lua elements; I’m trying to find alternatives that will allow it to work with the 5.1 Lua install on a Vera Home Automation controller (a relatively closed system).
Also, and perhaps most important for me is that I’d love to make as much of the converted code have minimal requirements to call external modules. I should add I’ve only recently started learning Lua, but one way I like to learn is to convert/repurpose code I find online..
So far i’ve managed to find alternatives for a number of the modules being used, e.g
encdec.base64dec -> Lua Base64 Encode
lmcore.hextostr -> https://github.com/tst2005/binascii/blob/master/binascii.lua
storage.set -> Alternative found in Vera Home Controllers
storage.get -> Alternative found in Vera Home Controllers
bit.ban -> Bitware module in Vera Home Controllers
bit.bxor -> Bitware module in Vera Home Controllers
Where I’m stuck is with the following..
aes:new
aes.cipher
user.aes
encdec.hmacsha256
Here’s an extract of the code where the above are used.
function encrypt_soap_payload(data, key, hmac_key, iv)
payload = '000000000000'
n = #data
payload = payload .. string.char(bit.band(bit.rshift(n, 24), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 16), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 8), 0xFF))
payload = payload .. string.char(bit.band(n, 0xFF))
payload = payload .. data
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 1)
ciphertext = aes_cbc:encrypt(payload)
sig = encdec.hmacsha256(ciphertext, hmac_key, true)
encrypted_payload = encdec.base64enc(ciphertext .. sig)
return encrypted_payload
end
function decrypt_soap_payload(data, key, hmac_key, iv)
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 0)
decrypted = aes_cbc:decrypt(encdec.base64dec(data))
decrypted = string.gsub(string.sub(lmcore.strtohex(decrypted), 33), '%x%x', function(value) return string.char(tonumber(value, 16)) end)
return decrypted
end
I can get the the point where I can create the parameters for the payload encrypt request (example below), it’s the encryption/decryption I can do..
data="1234"
key="\\S„ßÍ}/Ìa5!"
hmac_key="¹jz¹2¸F\r}òcžÎ„ 臧.ª˜¹=¤µæŸ"
iv=" {¬£áæ‚2žâ3ÐÞË€ú "
I’ve found an aes.lua module online, but that requires loads of others modules most notably ffi.lua. Ideally I’d like to avoid using that. I also came across this aes128.lua but i’m not sure how that handles all the other parameters e.g cbc etc. Finally there’s this aes256ecb.lua script, could that be converted to aes 128 cbc and then used in the above?
Is anyone aware (or maybe has) a Lua script that can handle the aes cbc requirements above ?
Many thanks !
In the end I found out that I could do aes.cbc by calling openssl from the command line, e.g.
local payload = "ENTER HERE"
Local key = "ENTER HERE"
local iv = "ENTER HERE"
local buildsslcommand = "openssl enc -aes-128-cbc -nosalt -e -a -A "..payload.." -K "..key.." -iv "..iv
-- print("Command to send = " ..buildsslcommand)
local file = assert(io.popen(buildsslcommand, 'r'))
local output = file:read('*all')
file:close()
-- print(string.len(output)) --> just count what's returned.
-- print(output) -- > Prints the output of the command.
FYI - It looks like I could do encdec.hmacsha256 via openSSL as well, but I’ve not been able to do that :-( ..
I don't use Lua, but need to use it with Openresty (nginx) as provided in the link.
Openresty has a lua module which I managed to install and run the Openresty nginx version correctly, the website working.
This answer shows how to concatenate headers into a string $request_headers:
set_by_lua $request_headers '
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
end
return request_headers_all
';
I changed the format from ""..k..": "..v..";" to "["..k..": "..v.."]" in the lua function above.
Log format:
log_format log_realip 'Host: "$host", Via : "$http_via", RemoteAddr: "$remote_addr", XForwardedFor: "$h
ttp_x_forwarded_for", 'RealIPRemoteAddr: "$realip_remote_addr" - $status - "$request_uri" - **"[HEADERS]" - $request_headers';**
Host: "192.168.1.4", Via : "-",
//trimmed just to show the [HEADERS]
....
"[HEADERS]" - [sec-ch-ua: \x22Chromium\x22;v=\x2288\x22, \x22Google Chrome\x22;v=\x228
8\x22, \x22;Not A Brand\x22;v=\x2299\x22][sec-ch-ua-mobile: ?0][cookie: __utmz=abcdef; frontend=abcdef; adminhtml=abcdef
08; TestName=Some Value][upgrade-insecure-requests: 1][accept-language: en-US,en;q=0.9][user-agent: Mozilla/5.0
(Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36][accept
-encoding: gzip, deflate, br][accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/we
bp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9][sec-fetch-dest: document][host: 192.168.1.4][se
c-fetch-user: ?1][connection: keep-alive][sec-fetch-mode: navigate][cache-control: max-age=0][sec-fetch-site: n
one
When using log_format with $request_headers string I get all the headers in one line, but I am trying to create a newline \n to break the string into lines. The example above is where I added \n but doesn't seem to output break to the log file.
I understand the request_headers_all .. concatenates the string, but what is happening here with the key k and value v : ""..k..": "..v..\n""?
What is the "..variablename.." doing, is this how variables are always used inside Lua strings?
How would I be able to create a line break in that string? Or is it possible that nginx(openresty) doesn't output the newline?
you add the \n to a wrong place, you can change to
request_headers_all = request_headers_all .. "["..k..": "..v.."]\n" for a newline log.
In lua, the .. is a concat operator, to concat to strings, for example:
print("hello" .. "world")
get the result helloworld.
your code \n"]" have syntax error, because \n not in a string.
lua strings can not directly use variables, usually, lua use string.format for complex string. for example:
local test = "hello"
string.format("%s world",test) -- hello world
you can use string.format for you string concat.
also, you can use table.concat to concat strings.
for example:
local test = {}
table.insert(test, "hello")
table.insert(test, "world")
local concated_string = table.concat(test, ' ')
print(concated_string) -- hello world
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
contains a syntax error. replace "["..k..": "..v..\n"]" with "["..k..": "..v.."]\n"
Newline needs to be inside the quotes as it is part of the string and it will probably make sense to add the new line after the bracket.
What is the "..variablename.." doing, is this how variables are always
used inside Lua strings?
using the concatenation operator on a variable concatenates a string value, concatenates its string representation if it is a number or invokes __concat or raises an error if neither of those is true.
Read https://www.lua.org/manual/5.4/manual.html#3.4.6
The above answers gave me some guidance, but the formats suggested still didn't work. After playing around with string.format("%s %s\n", k, v), string.format("%s %s\\n", k, v) I still got unfinished string errors or no newline output. (Tried to escape string in second example).
Based on the answers given I assumed the answers gave correct lua information, so decided most likely lua + openresty does something different.
I will attempt to update the title to reflect more specific requirements
TLDR
Openresty + lua string manipulation with special characters might not work as expected, even when using string.format()
Change from set_by_lua which returns a string to set_by_lua_block which allows string.format() or string concatenation better.
Update nginx configuration with your custom/existing log_format and add the switch escape=none.
Full Explanation
Investigating the provided link answer set_by_lua function documentation :
NOTE Use of this directive is discouraged following the v0.9.17 release. Use the set_by_lua_block directive instead.
So from the original set_by_lua from the link:
set_by_lua $request_headers '
return 'stringvalue'
';
I changed to set_by_lua_block function:
this directive inlines the Lua source directly inside a pair of curly braces ({}) instead of in an Nginx string literal (which requires special character escaping)
set_by_lua_block $request_headers{
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
local rowtext = ""
rowtext = string.format("[%s %s]\n", k, v)
request_headers_all = request_headers_all .. rowtext
end
return request_headers_all
}
The important part is that this _block {} function escapes the special characters correctly.
After that I received output in the log files as : x0A (newline character literal).
The final step then is to update the nginx.conf file with the custom log_format and add escape=none:
log_format log_realip escape=none "$nginx_variables"
The main question/problem here is how does the wrapper software handle newline characters/escapes, is it expecting "\n"? or is it expecting "\r\n"?
Ultimately the new line does not actually exist until it is interpreted and printed and you are creating one massive string that gets returned from the Lua engine to the wrapper, so it is up to the wrapper software on how to interpret new lines.
Edit: I missed that this was already answered by using the other parsing function.
Additionally the docs state to use the original parsing function escapes need to be double escaped.
Here, \\\\d+ is stripped down to \\d+ by the Nginx config file parser and this is further stripped down to \d+ by the Lua language parser before running.
I have next nginx location with Lua code (used with nginx lua-resty-redis):
location = /healthcheck {
content_by_lua_block {
local red = redis:new()
used_mem_limit = 536870912
-- connect to redis
ok, err = red:connect(ngx.var.redis_host, 6379)
if not ok then
ngx.log(ngx.ERR, "Redis connection error: ", err)
ngx.status = 500
return
end
-- setting connection timeout
red:set_timeout(1000)
-- getting redis used_memory_rss
memory = red:info('memory')
red_used_memory = ...
-- set response code
if (red_used_memory >= used_mem_limit) then
http_code = 500
elseif (red_used_memory < used_mem_limit)
http_code = 200
else
http_code = 500
end
ngx.status = http_code
}
}
memory variable should be like this:
# Memory
used_memory:105157968
used_memory_human:100.29M
used_memory_rss:110387200
used_memory_rss_human:105.27M
used_memory_peak:105219456
used_memory_peak_human:100.35M
used_memory_peak_perc:99.94%
used_memory_overhead:32290608
used_memory_startup:487168
used_memory_dataset:72867360
used_memory_dataset_perc:69.62%
total_system_memory:1044770816
I need to get used_memory_rss value, so then I can compare it with used_mem_limit. So, I don't know, how to complete red_used_memory = ... line.
There seems to be only digits in possible value so use a simple regexp and then convert resulting string to number:
red_used_memory = tonumber(memory:match('used_memory_rss:(%d+)'))
Line breaks are irrelevant.
Having trouble with file output in Nginx + Lua. I chosen LUA, because nginx logic is pretty complicated, based on referrer or subdomains, etc.
Having request like /img/am1/s/1.jpg I need to check if file exists in /somepath/am1/1.jpg. If it exists, then output it, otherwise proxy request to backend.
Ok, found it
content_by_lua '
local file = "/path..."
local f = io.open(file, "rb")
local content = f:read("*all")
f:close()
ngx.print(content)
';
If someone need to know how to output last n lines from file:
location /service-man/log {
default_type 'text/plain';
content_by_lua '
local log_path = "/path/to/log.log"
-- Opens a file in read
file = io.open(log_path, "r")
if file==nil
then
ngx.say(log_path .. " can\'t read or does not exists")
return
end
-- sets the default input file
io.input(file)
local lines = {}
-- read the lines in table lines
for line in io.lines() do
table.insert(lines, line)
end
io.close(file)
log_limit = 10
if #lines < log_limit then
log_start = 0
else
log_len = #lines
log_start = log_len - log_limit
end
local one_line = ""
for i, line in ipairs(lines) do
if i > log_start then
one_line = one_line .. line .. "\\n"
end
end
ngx.say(one_line)
';
}
Should be compatible with nginx/1.6.2 and Lua 5.3.
Please share if you know how to make it in more optimal way.