How to get client ip address in HttpHandler (Julia Language)? - http

I need to get client's IP address in regular HttpHandler like this:
http = HttpHandler() do req::Request, res::Response
Response( ismatch(r"^/hello/",req.resource) ? string("Hello ", split(req.resource,'/')[3], "!") : 404 )
end
Neither req nor http.sock contain this information.

The Approach
This can be done, if you know the internals of Julia a little. It turns out that Julia uses the library libuv for low level system processing and that library has a function called uv_tcp_getpeername. This function is not exported by Julia.Base, but you can gain access to it via ccall. Also, the module HttpServer allows for way define a callback for various events, including the connect event.
Example Module
module HTTPUtil
export get_server
using HttpServer
function handle_connect(client)
try
buffer = Array(Uint8,32)
bufflen::Int64 = 32
ccall(:uv_tcp_getpeername,Int64,(Ptr{Void},Ptr{Uint8},Ptr{Int64}),client.sock.handle,buffer,&bufflen)
peername::IPv4 = IPv4(buffer[5:8]...)
task_local_storage(:ip,peername)
catch e
println("Error ... $e")
end
end
function get_server()
http = HttpHandler() do req::Request, res::Response
ip = task_local_storage(:ip)
println("Connection received from from $ip")
Response(ismatch(r"^/hello/",req.resource)?string("Hello ",split(req.resource,'/')[3], " $(ip)!") : 404 )
end
http.events["connect"]=(client)->handle_connect(client)
server = Server(http)
end
end
Explained
Each time a connection request is made, a peer socket is created by the server, and the connect handler is called which is defined to be handle_connect. It takes one parameter, client, of type Client. A Client type has a field called sock of type TcpSocket, and a TcpSocket has a field handle which is used by libuv. The object, then is each time a connection request is made, the connect handler is called, which calls uv_tcp_getpeername with the data contained in the TcpSocket handle. A byte array is declared to act as a buffer, which then is cast back to Base.IPv4. The module HTTPServer creates exactly 1 task for each client using #async, so the ip address can be stored local to the client using task_local_storage; thus there is no race condition.
Using it
julia> using HTTPUtil
julia> server = get_server()
Server(HttpHandler((anonymous function),TcpServer(init),Dict{ASCIIString,Function} with 3 entries:
"error" => (anonymous function)
"listen" => (anonymous function)
"connect" => (anonymous function)),nothing)
julia> #async run(server,8000)
Listening on 8000...
Task (queued) #0x000000000767e7a0
julia> Connection received from from 192.168.0.23
Connection received from from 192.168.0.22
... etc
Notes
For illustration, the output is modified so that the server will respond to each browser "Hello ipaddr"
This should be included in Base and/or HttpServer, but currently is not, so you'll need to use this workaround until it is.
The typical looping structure is used in get_server to illustrate there is no requirement for it to change, except to add in the ip address.
Assumes IPv4, but can be improved to allow both IPv4 and IPv6 straightforwardly as libuv supports both.

Thanks to waTeim's answer, but it's from 2014 and things changed in Julia. This works nicely in Julia 6.0 and probably all above:
function ip(socket::TCPSocket)
buffer = Array{UInt8}(32)
bufflen::Int64 = 32
ccall(:uv_tcp_getpeername,Int64,(Ptr{Void},Ptr{UInt8},Ptr{Int64}), socket.handle, buffer, &bufflen)
peername::IPv4 = IPv4(buffer[5:8]...)
end

Building on the excellent answer from waTeim I simplified things a little to work with IPv6, and also for SSL connections:
using MbedTLS
function handle_connect(client)
ip, port = getsockname(isa(client.sock, MbedTLS.SSLContext) ? client.sock.bio : client.sock)
task_local_storage(:ip, ip)
end

(Would have added this as a comment to Josh Bode's answer, but I don't have the necessary reputation.)
Note that it is necessary to use getpeername() instead of getsockname() as of Julia v0.7.
https://github.com/JuliaLang/julia/pull/21825

Depending on your situation, you may be able to pass a linux command.
userIP = strip(readstring(`hostname -i`), ['\n'])

Related

Documentation for Airflow HTTP Operator/Sensor Extra Options?

I'm trying to read into the extra_options setting for Airflow to see what properties are possible to set (mainly interested in http timeout). I can't find any supporting documentation for this specific param anywhere: https://airflow.readthedocs.io/en/1.9.0/code.html?highlight=requests#airflow.operators.SimpleHttpOperator.
Has anyone worked with this before and is able to help?
According to source code (airflow.hooks.http_hook.HttpHook.run_and_check) extra_options uses these parameters:
response = session.send(
prepped_request,
stream=extra_options.get("stream", False),
verify=extra_options.get("verify", False),
proxies=extra_options.get("proxies", {}),
cert=extra_options.get("cert"),
timeout=extra_options.get("timeout"),
allow_redirects=extra_options.get("allow_redirects", True))
You can read more about them in requests library docs:
stream – (optional) whether to immediately download the response content. Defaults to False.
verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Defaults to True.
proxies – (optional) Dictionary mapping protocol or protocol and hostname to the URL of the proxy.
cert – (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert’, ‘key’) pair.
timeout (float or tuple) – (optional) How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.
allow_redirects (bool) – (optional) Set to True by default.
Following this trail of links in Airflow's source-code, you can easily determine what all things can be passed in SimpleHttpOperator, or more specifically, in extra field of Http Connection. I'm hereby adding trail of calls in Airflow's source that I used to trace the usage of extra_options
extra_options are passed to run() method of HttpHook
run() method of HttpHook passes those extra_options to run_and_check() method
run_and_check() method extracts variety of information from extra_options, as shown in the source-code snippet below
try:
response = session.send(
prepped_request,
stream=extra_options.get("stream", False),
verify=extra_options.get("verify", False),
proxies=extra_options.get("proxies", {}),
cert=extra_options.get("cert"),
timeout=extra_options.get("timeout"),
allow_redirects=extra_options.get("allow_redirects", True))
if extra_options.get('check_response', True):
self.check_response(response)
return response
except requests.exceptions.ConnectionError as ex:
self.log.warning(str(ex) + ' Tenacity will retry to execute the operation')
raise ex

AMI Asterisk Manager Interface Originate Issues with Outbound calls

I have a puzzling issue with my AMI implementation (totally new to this). I can successfully initiate calls to other extensions on my network. However, when I make outbound calls, they fail. I've installed Elastix using Asterisk 1.8.7.0 and FreePBX 2.8.1. On the outbound call, all I get is "goodbye" once the call has been placed. It successfully places the call but somehow it's not allowed by Asterisk?
Also note that all calls work well when using any SIP phone both to internal and external destinations! Therefore must be something I'm missing in the AMI implementation.
My manager.conf:
[user]
secret = 1111
deny=0.0.0.0/0.0.0.0
permit=192.168.0.0/255.255.255.0
read = all
write = all
Gave all permissions for read and write just to try and catch this one
My code (using Asterisk.NET):
Dim originate As New Asterisk.NET.Manager.Action.OriginateAction
originate.Channel = "SIP/106" 'This is the originating extension
originate.Context = "default" 'other option is - from-internal
originate.Exten = 7859855452
originate.Priority = 1
originate.CallerId = "106" 'This is the originating extension number
originate.Async = True
originate.Timeout = 30000
Dim response As New Asterisk.NET.Manager.Response.ManagerResponse
response = manager.SendAction(originate)
As per latest updates in Asterisk 1.8 if you are originating on external peer/trunk then channel variable will be like SIP/peer/extension instead of SIP/extension#peer

Serving non-standard HTTP method with ExpressJS

I would like to write an HTTP server that answer to request using a non-standard HTTP method (verb). For instance, the client would make a request like FOO / HTTP/.1.1. And on the server side, this request would be handled by something like:
var express = require('express');
var app = express.createServer();
app.configure(function(){
app.use(express.logger({ format: ':method :url' }));
app.use(express.methodOverride());
});
app.foo('/', function(req, res){
res.send('Hello World');
});
app.listen(3000);
I appended my non-standard method to the array exported in ExpressJS's lib/router/methods.js. This allow me to write my server code as expected. When using express.methodOverride() and a POST request with _method=foo, it works. But an actual FOO request doesn't work. As soon as the client send the first line of the request the connection is closed by the server:
$telnet localhost 3000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
FOO / HTTP/1.1
Connection closed by foreign host.
I would like to be able to implement this with ExpressJS and without avoid hacking into its core file.
Any idea if this is possible and how?
Short answer: No, it's not possible. Not without implementing your own HTTP module.
To test, start a barebones HTTP server ...
$ node
> require('http').createServer(function(req, res) {
... console.log(req.method);
... res.end();
... }).listen(8080);
Then (as you've already done) telnet to it and issue a GET and FOO request ...
$ telnet localhost 8080
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked
0
FOO / HTTP/1.1
Connection closed by foreign host.
$
In node console you'll see
GET
... but no FOO. So, node's native HTTP module, which Express uses, does not make these requests available.
Node has a hard-coded whitelist of acceptable HTTP verbs in C.
In order to accept custom verbs, you must modify the HTTP parser and recompile node.
You mentioned that you're trying to implement PURGE, which was added to the whitelist in v0.7.5.
As others have said, Node.js' HTTP server library is configured to accept only specific verbs. Ben Noordius' suggestion of using Parsley doesn't work either, since that library accepts an even smaller whitelist of verbs. (It also hasn't been maintained in quite some time.)
At this stage, if we want to support oddball requests, we have to take more drastic measures. Here's a nice ugly hack for you that involves duck punching some internal behavior. This works on v0.10.x of Node.js, but test carefully on newer versions as they become available.
In my case, I needed to support not only a non-standard verb, but a non-standard protocol version identifier as well, and a missing Content-Length header for Icecast source streams:
SOURCE /live ICE/1.0
The following should get you started:
server.on('connection', function (socket) {
var originalOnDataFunction = socket.ondata;
var newLineOffset;
var receiveBuffer = new Buffer(0);
socket.ondata = function (d, start, end) {
receiveBuffer = Buffer.concat([receiveBuffer, d.slice(start, end)]);
if ((newLineOffset = receiveBuffer.toString('ascii').indexOf('\n')) > -1) {
var firstLineParts = receiveBuffer.slice(0, newLineOffset).toString().split(' ');
firstLineParts[0] = firstLineParts[0].replace(/^SOURCE$/ig, 'PUT');
firstLineParts[2] = firstLineParts[2].replace(/^ICE\//ig, 'HTTP/');
receiveBuffer = Buffer.concat([
new Buffer(
firstLineParts.join(' ') + '\r\n' +
'Content-Length: 9007199254740992\r\n'
),
receiveBuffer.slice(newLineOffset +1)
]);
socket.ondata = originalOnDataFunction;
socket.ondata.apply(this, [receiveBuffer, 0, receiveBuffer.length]);
}
};
}
It's ugly, but works. I'm not particularly happy about it, but when choosing between a rough built-from-the-ground-up HTTP parser or tweaking an existing one, I choose to tweak in this instance.
For anyone who needs it, there is http-parser-js, which replaces Node's built-in HTTP parser.
Their README contains an example of monkey-patching the parser, though I find that it wasn't enough, as both the http-parser-js and the http modules have a hardcoded list of methods.
So, you have to replace the parser and edit the list of methods:
const { HTTPParser } = require('http-parser-js');
HTTPParser.methods.push('FOOBAR');
const binding = process.binding('http_parser');
binding.HTTPParser = HTTPParser;
binding.methods = HTTPParser.methods;
require('http').METHODS = HTTPParser.methods;
Later Node versions may not support process.binding, in which case, you can use the --expose-internals flag for Node (see this issue):
const { internalBinding } = require('internal/test/binding');
const binding = internalBinding('http_parser');
From the looks of it, the http2 module's parser accepts any method, in case that's an option. See this issue about invalid HTTP methods. Unfortunately, express and the likes do not use http2.
And for anyone who was in my shoes, proxying requests to a legacy server in Create React App, use the above snippet in webpack-dev-server, at the top of Server.js, in order to monkey-patch the parser. Hopefully everything switches to http2 soon...

How do I log asynchronous thin+sinatra+rack requests?

I'm writing my first Sinatra-based web app as a frontend to another TCP-based service, using EventMachine and async_sinatra to process incoming HTTP requests asynchronously. When I'm testing my app, all requests to synchronous routes are logged to stdout in common log format, but asynchronous requests are not.
I've read through bits of the source code to async_sinatra, Sinatra, Thin, and Rack, and it looks like logging of synchronous requests is done through CommonLogger#call. However, I can't find anywhere in the asynchronous code in async_sinatra or Thin that seems to pass asynchronous requests through the logging middleware (I'm looking at Sinatra::Helpers#body in async_sinatra and at Thin::Connection.post_process which is written into env['.async_callback'] in Thin's connection.rb:68 and request.rb:132).
I'm experienced with C but relatively new to Ruby, so if I've used some terminology or notation incorrectly, please correct me. Thanks in advance.
Edit: this also affects error handling. If an exception is raised in an asynchronous request, the request is never finished and the error is never logged.
I eventually found that using rack-async with async_sinatra was causing problems with 404 pages, exception handling, and logging:
!! Unexpected error while processing request: undefined method `bytesize' for nil:NilClass
Instead I used the following wrapper around aroute for logging:
module Sinatra::Async
alias :oldaroute :aroute
def aroute verb, path, opts = {}, &block
# Based on aroute from async_sinatra
run_method = :"RunA#{verb} #{path} #{opts.hash}"
define_method run_method, &block
log_method = :"LogA#{verb} #{path} #{opts.hash}"
define_method(log_method) { |*a|
puts "#{request.ip} - #{status} #{verb} #{path}"
}
oldaroute verb, path, opts do |*a|
oldcb = request.env['async.callback']
request.env['async.callback'] = proc { |*args|
async_runner(log_method, *a)
oldcb[*args]
}
async_runner(run_method, *a)
end
end
end
This is for the same versions of async_sinatra, Thin, and Rack that I was using when I asked this question last year; newer versions may allow the use of common Rack middleware for logging.
I am running on sinatra-synchrony and therefore I have a slightly different core than you.
But basically I solved the same problem.
Here is an abstract of the solution:
I am not using Rack::CommonLogger, I use my own Logger
You need to buffer log output in an async aware storage
The buffered log output must be flushed at the end of the request
In my sinatra-synchrony application I am running the following middleware for logging:
# in app.rb I register Logger::Middleware as the first middleware
use Logger::Middleware
# in logger.rb
module Logger
attr_accessor :messages
def log(message)
stack << message
end
def stack
# This is the important async awareness
# It stores messages for each fiber separately
messages[Fiber.current.object_id] ||= []
end
def flush
STDERR.puts stack.join("\n") unless stack.empty?
messages.delete Fiber.current.object_id
end
extend self
class Middleware
def initialize(app)
#app = app
end
def call(env)
# before the request
Logger.log "#{env['REQUEST_METHOD']} #{env['REQUEST_URI']}"
result = #app.call(env)
# after the request
Logger.flush
result
end
end
end
Logger.messages = {} # initialize the message storage
Everywhere in the application I am able to use Logger.log("message") for logging.

HTTP POST request in Inno Setup Script

I would like to submit some information collected from user during Inno setup installation to our server via POST.
Obvious solution would be to include an .exe file that the setup would extract into temporary location and launch with parameters. However, I'm wondering - is there is any easier/better way?
Based on jsobo advice of using WinHTTP library, I came with this very simple code that does the trick. Say, you want to send serial number for verification just before the actual installation starts. In the Code section, put:
procedure CurStepChanged(CurStep: TSetupStep);
var
WinHttpReq: Variant;
begin
if CurStep = ssInstall then
begin
if AutoCheckRadioButton.Checked = True then
begin
WinHttpReq := CreateOleObject('WinHttp.WinHttpRequest.5.1');
WinHttpReq.Open('POST', '<your_web_server>', false);
WinHttpReq.SetRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
WinHttpReq.Send('<your_data>');
{ WinHttpReq.ResponseText will hold the server response }
end;
end;
end;
The Open method takes as arguments the HTTP method, the URL and whether to do async request and it seems like we need to add SetRequestHeader in order to set the Content-Type header to application/x-www-form-urlencoded.
WinHttpReq.Status will hold the response code, so to check if the server returned successfully:
if WinHttpReq.Status <> 200 then
begin
MsgBox('ERROR', mbError, MB_OK);
end
else
begin
MsgBox('SUCCESS', mbInformation, MB_OK);
end;
https://learn.microsoft.com/en-us/windows/win32/winhttp/winhttprequest lists all methods and properties of the WinHttpRequest object.
Also, to avoid run-time errors (can happen if the host is unreachable) it is a good idea to surround the code with try/except code.
You could always have your installer use curl
to make the http post...
You could write a pascal script right in innosetup to do the call utilizing the winhttp library
Or you could just write a vbscript and execute that with the cscript engine to do the same http call via the winhttp library.
That should point you to at least 3 different options to do what you need.
I think putting the exe in there would be the least error prone but utilizing the winhttp library with the pascal script (used by innosetup) would be the most simple.
I haven't tried it but the ISXKB has an entry for an uninstall survey that uses an HTTP POST:
http://www.vincenzo.net/isxkb/index.php?title=Uninstall_Survey

Resources