I have two simple functions that detects browser and operating system based on user agent and they are stored in file useragent.lua.
function detect_browser_platform(user_agent)
-- Here goes some string matching and similar stuff
return browser_platform
end
function detect_os_platform(user_agent)
-- Here goes some string matching and similar stuff
return os_platform
end
function detect_env_pattern(user_agent)
return detect_operating_system_platform(user_agent) .. "-" .. detect_browser_platform(user_agent) .. "-" .. ngx.var.geoip2_data_country_code
end
In virtual host configuration file, there is a line that says when request looks like /lua execute lua script: /var/www/default/test.lua.
In test.lua I have this code:
local posix = require('posix')
local redis = require('redis')
require('useragent')
-- Some code goes here
local user_agent = ngx.req.get_headers()['User-Agent']
local pattern_string = detect_env_pattern(user_agent)
ngx.say(pattern_string)
ngx.exit(200)
But for some reason when I reload nginx nginx -s reload, this codes works only first time. When I make another request it says this error:
2016/09/19 12:30:08 [error] 19201#0: *125956 lua entry thread aborted: runtime error: /var/www/default/test.lua:199: attempt to call global 'detect_env_pattern' (a nil value)
And I have no idea what is happening here. I have just started programming in Lua and don't have time to go deep with language understandings... So why am I getting this error?
Wrap it by a table:
local M={};
function detect_browser_platform(user_agent)
-- Here goes some string matching and similar stuff
return browser_platform
end
function detect_os_platform(user_agent)
-- Here goes some string matching and similar stuff
return os_platform
end
function detect_env_pattern(user_agent)
return detect_operating_system_platform(user_agent) .. "-" .. detect_browser_platform(user_agent) .. "-" .. ngx.var.geoip2_data_country_code
end
M.detect_env_pattern = detect_env_pattern
return M
in base lua file:
local useragent = require('useragent')
--.....
local user_agent = ngx.req.get_headers()['User-Agent']
local pattern_string = useragent.detect_env_pattern(user_agent)
ngx.say(pattern_string)
ngx.exit(200)
Related
i'm new to nf-core/nextflow and needless to say the documentation does not reflect what might be actually implemented. But i'm defining the basic pipeline below:
nextflow.enable.dsl=2
process RUNBLAST{
input:
val thr
path query
path db
path output
output:
path output
script:
"""
blastn -query ${query} -db ${db} -out ${output} -num_threads ${thr}
"""
}
workflow{
//println "I want to BLAST $params.query to $params.dbDir/$params.dbName using $params.threads CPUs and output it to $params.outdir"
RUNBLAST(params.threads,params.query,params.dbDir, params.output)
}
Then i'm executing the pipeline with
nextflow run main.nf --query test2.fa --dbDir blast/blastDB
Then i get the following error:
N E X T F L O W ~ version 22.10.6
Launching `main.nf` [dreamy_hugle] DSL2 - revision: c388cf8f31
Error executing process > 'RUNBLAST'
Error executing process > 'RUNBLAST'
Caused by:
Not a valid path value: 'test2.fa'
Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run
I know test2.fa exists in the current directory:
(nfcore) MN:nf-core-basicblast jraygozagaray$ ls
CHANGELOG.md conf other.nf
CITATIONS.md docs pyproject.toml
CODE_OF_CONDUCT.md lib subworkflows
LICENSE main.nf test.fa
README.md modules test2.fa
assets modules.json work
bin nextflow.config workflows
blast nextflow_schema.json
I also tried with "file" instead of path but that is deprecated and raises other kind of errors.
It'll be helpful to know how to fix this to get myself started with the pipeline building process.
Shouldn't nextflow copy the file to the execution path?
Thanks
You get the above error because params.query is not actually a path value. It's probably just a simple String or GString. The solution is to instead supply a file object, for example:
workflow {
query = file(params.query)
BLAST( query, ... )
}
Note that a value channel is implicitly created by a process when it is invoked with a simple value, like the above file object. If you need to be able to BLAST multiple query files, you'll instead need a queue channel, which can be created using the fromPath factory method, for example:
params.query = "${baseDir}/data/*.fa"
params.db = "${baseDir}/blastdb/nt"
params.outdir = './results'
db_name = file(params.db).name
db_path = file(params.db).parent
process BLAST {
publishDir(
path: "{params.outdir}/blast",
mode: 'copy',
)
input:
tuple val(query_id), path(query)
path db
output:
tuple val(query_id), path("${query_id}.out")
"""
blastn \\
-num_threads ${task.cpus} \\
-query "${query}" \\
-db "${db}/${db_name}" \\
-out "${query_id}.out"
"""
}
workflow{
Channel
.fromPath( params.query )
.map { file -> tuple(file.baseName, file) }
.set { query_ch }
BLAST( query_ch, db_path )
}
Note that the usual way to specify the number of threads/cpus is using cpus directive, which can be configured using a process selector in your nextflow.config. For example:
process {
withName: BLAST {
cpus = 4
}
}
I’m trying to rework a script I found online to control a Panasonic TV, which requires a secure/encrypted pairing to occur so I can control it remotely. (The full code here -> https://forum.logicmachine.net/showthread.php?tid=232&pid=16580#pid16580)
Because it seems to be built on LuaJIT and has some other proprietary Lua elements; I’m trying to find alternatives that will allow it to work with the 5.1 Lua install on a Vera Home Automation controller (a relatively closed system).
Also, and perhaps most important for me is that I’d love to make as much of the converted code have minimal requirements to call external modules. I should add I’ve only recently started learning Lua, but one way I like to learn is to convert/repurpose code I find online..
So far i’ve managed to find alternatives for a number of the modules being used, e.g
encdec.base64dec -> Lua Base64 Encode
lmcore.hextostr -> https://github.com/tst2005/binascii/blob/master/binascii.lua
storage.set -> Alternative found in Vera Home Controllers
storage.get -> Alternative found in Vera Home Controllers
bit.ban -> Bitware module in Vera Home Controllers
bit.bxor -> Bitware module in Vera Home Controllers
Where I’m stuck is with the following..
aes:new
aes.cipher
user.aes
encdec.hmacsha256
Here’s an extract of the code where the above are used.
function encrypt_soap_payload(data, key, hmac_key, iv)
payload = '000000000000'
n = #data
payload = payload .. string.char(bit.band(bit.rshift(n, 24), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 16), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 8), 0xFF))
payload = payload .. string.char(bit.band(n, 0xFF))
payload = payload .. data
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 1)
ciphertext = aes_cbc:encrypt(payload)
sig = encdec.hmacsha256(ciphertext, hmac_key, true)
encrypted_payload = encdec.base64enc(ciphertext .. sig)
return encrypted_payload
end
function decrypt_soap_payload(data, key, hmac_key, iv)
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 0)
decrypted = aes_cbc:decrypt(encdec.base64dec(data))
decrypted = string.gsub(string.sub(lmcore.strtohex(decrypted), 33), '%x%x', function(value) return string.char(tonumber(value, 16)) end)
return decrypted
end
I can get the the point where I can create the parameters for the payload encrypt request (example below), it’s the encryption/decryption I can do..
data="1234"
key="\\S„ßÍ}/Ìa5!"
hmac_key="¹jz¹2¸F\r}òcžÎ„ 臧.ª˜¹=¤µæŸ"
iv=" {¬£áæ‚2žâ3ÐÞË€ú "
I’ve found an aes.lua module online, but that requires loads of others modules most notably ffi.lua. Ideally I’d like to avoid using that. I also came across this aes128.lua but i’m not sure how that handles all the other parameters e.g cbc etc. Finally there’s this aes256ecb.lua script, could that be converted to aes 128 cbc and then used in the above?
Is anyone aware (or maybe has) a Lua script that can handle the aes cbc requirements above ?
Many thanks !
In the end I found out that I could do aes.cbc by calling openssl from the command line, e.g.
local payload = "ENTER HERE"
Local key = "ENTER HERE"
local iv = "ENTER HERE"
local buildsslcommand = "openssl enc -aes-128-cbc -nosalt -e -a -A "..payload.." -K "..key.." -iv "..iv
-- print("Command to send = " ..buildsslcommand)
local file = assert(io.popen(buildsslcommand, 'r'))
local output = file:read('*all')
file:close()
-- print(string.len(output)) --> just count what's returned.
-- print(output) -- > Prints the output of the command.
FYI - It looks like I could do encdec.hmacsha256 via openSSL as well, but I’ve not been able to do that :-( ..
I don't use Lua, but need to use it with Openresty (nginx) as provided in the link.
Openresty has a lua module which I managed to install and run the Openresty nginx version correctly, the website working.
This answer shows how to concatenate headers into a string $request_headers:
set_by_lua $request_headers '
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
end
return request_headers_all
';
I changed the format from ""..k..": "..v..";" to "["..k..": "..v.."]" in the lua function above.
Log format:
log_format log_realip 'Host: "$host", Via : "$http_via", RemoteAddr: "$remote_addr", XForwardedFor: "$h
ttp_x_forwarded_for", 'RealIPRemoteAddr: "$realip_remote_addr" - $status - "$request_uri" - **"[HEADERS]" - $request_headers';**
Host: "192.168.1.4", Via : "-",
//trimmed just to show the [HEADERS]
....
"[HEADERS]" - [sec-ch-ua: \x22Chromium\x22;v=\x2288\x22, \x22Google Chrome\x22;v=\x228
8\x22, \x22;Not A Brand\x22;v=\x2299\x22][sec-ch-ua-mobile: ?0][cookie: __utmz=abcdef; frontend=abcdef; adminhtml=abcdef
08; TestName=Some Value][upgrade-insecure-requests: 1][accept-language: en-US,en;q=0.9][user-agent: Mozilla/5.0
(Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36][accept
-encoding: gzip, deflate, br][accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/we
bp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9][sec-fetch-dest: document][host: 192.168.1.4][se
c-fetch-user: ?1][connection: keep-alive][sec-fetch-mode: navigate][cache-control: max-age=0][sec-fetch-site: n
one
When using log_format with $request_headers string I get all the headers in one line, but I am trying to create a newline \n to break the string into lines. The example above is where I added \n but doesn't seem to output break to the log file.
I understand the request_headers_all .. concatenates the string, but what is happening here with the key k and value v : ""..k..": "..v..\n""?
What is the "..variablename.." doing, is this how variables are always used inside Lua strings?
How would I be able to create a line break in that string? Or is it possible that nginx(openresty) doesn't output the newline?
you add the \n to a wrong place, you can change to
request_headers_all = request_headers_all .. "["..k..": "..v.."]\n" for a newline log.
In lua, the .. is a concat operator, to concat to strings, for example:
print("hello" .. "world")
get the result helloworld.
your code \n"]" have syntax error, because \n not in a string.
lua strings can not directly use variables, usually, lua use string.format for complex string. for example:
local test = "hello"
string.format("%s world",test) -- hello world
you can use string.format for you string concat.
also, you can use table.concat to concat strings.
for example:
local test = {}
table.insert(test, "hello")
table.insert(test, "world")
local concated_string = table.concat(test, ' ')
print(concated_string) -- hello world
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
contains a syntax error. replace "["..k..": "..v..\n"]" with "["..k..": "..v.."]\n"
Newline needs to be inside the quotes as it is part of the string and it will probably make sense to add the new line after the bracket.
What is the "..variablename.." doing, is this how variables are always
used inside Lua strings?
using the concatenation operator on a variable concatenates a string value, concatenates its string representation if it is a number or invokes __concat or raises an error if neither of those is true.
Read https://www.lua.org/manual/5.4/manual.html#3.4.6
The above answers gave me some guidance, but the formats suggested still didn't work. After playing around with string.format("%s %s\n", k, v), string.format("%s %s\\n", k, v) I still got unfinished string errors or no newline output. (Tried to escape string in second example).
Based on the answers given I assumed the answers gave correct lua information, so decided most likely lua + openresty does something different.
I will attempt to update the title to reflect more specific requirements
TLDR
Openresty + lua string manipulation with special characters might not work as expected, even when using string.format()
Change from set_by_lua which returns a string to set_by_lua_block which allows string.format() or string concatenation better.
Update nginx configuration with your custom/existing log_format and add the switch escape=none.
Full Explanation
Investigating the provided link answer set_by_lua function documentation :
NOTE Use of this directive is discouraged following the v0.9.17 release. Use the set_by_lua_block directive instead.
So from the original set_by_lua from the link:
set_by_lua $request_headers '
return 'stringvalue'
';
I changed to set_by_lua_block function:
this directive inlines the Lua source directly inside a pair of curly braces ({}) instead of in an Nginx string literal (which requires special character escaping)
set_by_lua_block $request_headers{
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
local rowtext = ""
rowtext = string.format("[%s %s]\n", k, v)
request_headers_all = request_headers_all .. rowtext
end
return request_headers_all
}
The important part is that this _block {} function escapes the special characters correctly.
After that I received output in the log files as : x0A (newline character literal).
The final step then is to update the nginx.conf file with the custom log_format and add escape=none:
log_format log_realip escape=none "$nginx_variables"
The main question/problem here is how does the wrapper software handle newline characters/escapes, is it expecting "\n"? or is it expecting "\r\n"?
Ultimately the new line does not actually exist until it is interpreted and printed and you are creating one massive string that gets returned from the Lua engine to the wrapper, so it is up to the wrapper software on how to interpret new lines.
Edit: I missed that this was already answered by using the other parsing function.
Additionally the docs state to use the original parsing function escapes need to be double escaped.
Here, \\\\d+ is stripped down to \\d+ by the Nginx config file parser and this is further stripped down to \d+ by the Lua language parser before running.
I have compiled nginx with Lua. and i am facing a problem .
When i hit localhost i get an error in nginx error logs
2015/05/29 07:41:32 [error] 112#0: *1 lua entry thread aborted: runtime error: /opt/hello.lua:11: bad argument #1 to 'getNewSessionID' (string expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'getNewSessionID'
Nginx configurations are
location / {
default_type 'text/html';
content_by_lua_file /opt/hello.lua;
}
And my hello.lua look like
package.path = package.path ..';/opt/?.lua'
JSON = (loadfile "/opt/JSON.lua")()
local proxyFunctions = require("proxyCode")
coherenceNGIXLocation = "/coherence"
local coherenceCookie = proxyFunctions.getExistingSessionID()
-- To Create a coherance Session if not present
if (not coherenceCookie or coherenceCookie == '') then
-- ngx.log(ngx.INFO,"No Coherence Session Exists, Creating New")
local uidCookieValueEncoded = proxyFunctions.base64Encode(str,proxyFunctions.getNewSessionID())
local sessCreateResponse = proxyFunctions.coherencePut('{"sessionToken":"'..uidCookieValueEncoded..'"}')
local parsedJSON= JSON:decode(sessCreateResponse.body)
ngx.log(ngx.INFO, "Coherence Session Create")
ngx.log(ngx.INFO, "\n\n\n=============Coherence Res==============\n\n\n"..sessCreateResponse.body.."\n\n\n=============Coherence Res==============\n\n\n")
end
local client_cookie = ngx.req.get_headers()["cookie"]
-- To Create a coherance Session if not present
ngx.log(ngx.DEBUG, "\n\n\n=============REQ==============\n\n\n"..proxyFunctions.deepPrint(ngx.req.get_headers()).."\n\n\n=============REQ==============\n\n\n")
-- Set the cookie for the legacy request
ngx.req.set_header('cookie', client_cookie)
ngx.log(ngx.INFO, "\n\n\n=============REQ URI==============\n\n\n"..ngx.var.request_uri.."\n\n\n=============REQ URI==============\n\n\n")
local legacy_res = ngx.location.capture("/app"..ngx.var.request_uri, { share_all_vars = true })
ngx.log(ngx.DEBUG, "\n\n\n=============APP Res==============\n\n\n"..proxyFunctions.deepPrint(legacy_res.header).."\n\n\n=============APP Res==============\n\n\n")
if (legacy_res.status == 302) then
ngx.redirect(legacy_res.header.Location )
else
ngx.log(ngx.INFO, "Passing the Page/Layout to Maken")
local dev_res_encoded = legacy_res.body
dev_res_encoded = ngx.escape_uri(dev_res_encoded)
-- ngx.req.set_header('Cookie', ngx.req.get_headers()["cookie"])
-- ngx.req.set_header("Content-Type", "text/html") -- application/x-www-form-urlencoded -- 'page='..dev_res_encoded
ngx.req.set_header('cookie', client_cookie) -- maken/render
-- legacy_res.header('Content-Length', legacy_res.header['Content-Length']);
local maken_res = ngx.location.capture("/maken/", {method = ngx.HTTP_POST, body = legacy_res.body, share_all_vars = true} )
ngx.header['Set-Cookie'] = legacy_res.header["Set-Cookie"]
ngx.log(ngx.INFO, "Sending Back Maken Response to Client/Browser with Cookies from lagacy")
ngx.say(maken_res.body)
end
My proxyCode.lua looks like
function _M.getNewSessionID()
-- strip the sessionID length 11 cookie name before putting into the session
-- return string.sub(ngx.var.uid_set,11)
return _M.removeCookieSessionPrefixname(ngx.var.uid_set,11)
end
function _M.removeCookieSessionPrefixname(cookieName)
return string.sub(cookieName,11)
end
function _M.getIncomingAuthToken()
local cookie = ngx.req.get_headers()["Cookie"]
if (cookie ~=nil) then
local str = string.match(cookie,"AuthToken=(.*)$")
if (str ~= nil) then
return string.sub(str,11)
end
end
return nil
end
Can someone help me why i am getting this error ? i am new to Lua programming. and trying to run legacy application on my laptop.
I'm trying to use haxr 3000.8.5 to upload images to a WordPress blog using the metaWeblog API---specifically, the newMediaObject method.
I've gotten it to work for small images, having successfully uploaded 20x20 icons in both PNG and JPG formats. However, when I try medium-sized images (say, 300x300) I get an ErrorClosed exception, presumably from the HTTP package (I did a bit of source diving, and found that haxr ultimately calls Network.HTTP.simpleHTTP).
Can anyone shed light on the reasons why a call to simpleHTTP might fail with ErrorClosed? Suggestions of things to try and potential workarounds are also welcome.
Here are links to full tcpdump output from a successful upload and from an unsuccessful upload.
The (sanitized) code is also shown below, in case it's of any use.
import Network.XmlRpc.Client (remote)
import Network.XmlRpc.Internals (Value(..), toValue)
import Data.Char (toLower)
import System.FilePath (takeFileName, takeExtension)
import qualified Data.ByteString.Char8 as B
import Data.Functor ((<$>))
uploadMediaObject :: FilePath -> IO Value
uploadMediaObject file = do
media <- mkMediaObject file
remote "http://someblog.wordpress.com/xmlrpc.php" "metaWeblog.newMediaObject"
"default" "username" "password" media
-- Create the required struct representing the image.
mkMediaObject :: FilePath -> IO Value
mkMediaObject filePath = do
bits <- B.unpack <$> B.readFile filePath
return $ ValueStruct
[ ("name", toValue fileName)
, ("type", toValue fileType)
, ("bits", ValueBase64 bits)
]
where
fileName = takeFileName filePath
fileType = case (map toLower . drop 1 . takeExtension) fileName of
"png" -> "image/png"
"jpg" -> "image/jpeg"
"jpeg" -> "image/jpeg"
"gif" -> "image/gif"
main = do
v <- uploadMediaObject "images/puppy.png"
print v
21:59:56.813021 IP 192.168.1.148.39571 > ..http: Flags [.]
22:00:01.922598 IP ..http > 192.168.1.148.39571: Flags [F.]
The connection is closed by the server after a 3-4 sec timeout as the client didn't send any data, to prevent slowloris and similar ddos attacks. (F is the FIN flag, to close one direction of the bidirectional connection).
The server does not wait for the client to close the connection (wait for eof/0 == recv(fd)) but uses the close() syscall; the kernel on the server will respond with [R]eset packets if it receives further data, as you can see at the end of your dump.
I guess the client first opens the http connection and then prepares the data which takes too long.