I'm trying to get the both the client request and IP address from http requests to my HTTP.jl server (based on the basic server example in the docs).
using HTTP
using Sockets
const APP = HTTP.Router()
# My request handler function can see the request's method
# and target but not the IP address it came from
HTTP.#register(APP,"GET","/",req::HTTP.Request -> begin
println("$(req.method) request to $(req.target)")
"Hello, world!"
end)
HTTP.serve(
APP,
Sockets.localhost,
8081;
# My tcpisvalid function can see the client's
# IP address but not the HTTP request
tcpisvalid=sock::Sockets.TCPSocket -> begin
host, port = Sockets.getpeername(sock)
println("Request from $host:$port")
true
end
)
My best guess would be that there's a way to parse the TCPSocket.buffer into an HTTP request but I can't find any methods to do it.
Can you suggest a way to get an HTTP.Request from a TCPSocket or a different way to approach this problem?
Thanks in advance!
The router (APP) is a (collection of) "request handler(s)" which can only access the HTTP.Request -- you can not get the stream from it. Instead you can define a "stream handler", which is passed the stream. From the stream you can get the client's IP adress using Sockets.getpeername (requires HTTP.jl version 0.9.7 when called on a HTTP.Stream as in the examples below).
using HTTP, Sockets
const APP = HTTP.Router()
function request_handler(req::HTTP.Request)
println("$(req.method) request to $(req.target)")
return "Hello, world!"
end
HTTP.#register APP "GET" "/" request_handler
function stream_handler(http::HTTP.Stream)
host, port = Sockets.getpeername(http)
println("Request from $host:$port")
return HTTP.handle(APP, http) # regular handling
end
# HTTP.serve with stream=true to specify that stream_handler is a function
# that expects a HTTP.Stream as input (and not a HTTP.Request)
HTTP.serve(stream_handler, Sockets.localhost, 8081; stream=true) # <-- Note stream=true
# or HTTP.listen
HTTP.listen(stream_handler, Sockets.localhost, 8081)
Related
I am trying to turn some Julia code into a very basic local microservice, which accepts a POST request with some options supplied via JSON, runs a program, and returns a 200 response. My microservice code is here:
const ROUTERS = Dict()
function __init__()
"""Initialisation function to populate the global variables."""
ROUTERS["ROUTER"] = HTTP.Router()
HTTP.register!(ROUTERS["ROUTER"], "POST", "/run", run_program)
end
function run_program(req)
"""Takes a run request and passes it to the main program, before sending a 200 reponse to say no errors have occured."""
setup = JSON3.read(req.body, Dict{String, Any})
run_program(setup)
return HTTP.Response(200, JSON3.write("Success."))
end
function requestHandler(req)
"""Recieves incoming requests and passes them to the router. Returns the response object."""
local resp
resp = HTTP.handle(ROUTERS["ROUTER"], req)
return resp
end
function run_service(port=8080)
"""Runs the microservice at the specified port."""
HTTP.serve(requestHandler, "0.0.0.0", port)
end
__init__()
This code works with HTTP version 0.9.17, but I updated it to the new version 1.5.5. Now I receive this error whenever I make a request to the running service:
LogLevel(1999): handle_connection handler error
│ exception =
│ UndefVarError: handle not defined
What am I missing here? Have I defined my handler function incorrectly somehow?
There's no HTTP.handle function anymore in HTTP.jl version 1.x, as outlined by the documentation.
You'll probably want something like
"""
Receives incoming requests and passes them to the router. Returns the response object.
"""
function requestHandler(req)
return ROUTERS["ROUTER"](req)
end
instead. Also note that docstrings need to be inserted before the function for the docsystem to pick them up.
I'm currently creating a go TCP server that handles file sharing between multiple go clients, that works fine. However, I'm also building a front end using vue.js showing some server stats like the number of users, bytes sent, etc.
The problem occurs when I include the 'http.ListenAndServe(":3000", nil)' function handles the requests from the front end of the server. Is it impossible to have a TCP and an HTTP server on the same go file?
If so, how can a link the three (frontend, go-server, clients)
Here is the code of the 'server.go'
func main() {
// Create TCP server
serverConnection, error := net.Listen("tcp", ":8085")
// Check if an error occured
// Note: because 'go' forces you to use each variable you declare, error
// checking is not optional, and maybe that's good
if error != nil {
fmt.Println(error)
return
}
// Create server Hub
serverHb := newServerHub()
// Close the server just before the program ends
defer serverConnection.Close()
// Handle Front End requests
http.HandleFunc("/api/thumbnail", requestHandler)
fs := http.FileServer(http.Dir("../../tcp-server-frontend/dist"))
http.Handle("/", fs)
fmt.Println("Server listening on port 3000")
http.ListenAndServe(":3000", nil)
// Each client sends data, that data is received in the server by a client struct
// the client struct then sends the data, which is a request to a 'go' channel, which is similar to a queue
// Somehow this for loop runs only when a new connection is detected
for {
// Accept a new connection if a request is made
// serverConnection.Accept() blocks the for loop
// until a connection is accepted, then it blocks the for loop again!
connection, connectionError := serverConnection.Accept()
// Check if an error occurred
if connectionError != nil {
fmt.Println("1: Woah, there's a mistake here :/")
fmt.Println(connectionError)
fmt.Println("1: Woah, there's a mistake here :/")
// return
}
// Create new user
var client *Client = newClient(connection, "Unregistered_User", serverHb)
fmt.Println(client)
// Add client to serverHub
serverHb.addClient(client)
serverHb.listClients()
// go client.receiveFile()
go client.handleClientRequest()
}
}
I am trying to implement a Reverse Proxy in Go that proxies traffic to different hosts based on some tenant embedded in the URL. The implementation looks like this:
type Offloader struct {
tenantHostMap map[string]string // Map a tenant to its host:port
tenantProxyMap map[string](*httputil.ReverseProxy) // Map a tenant to its reverse proxy
}
func (o *Offloader) OnCreate() {
// Tenants Map
o.tenantHostMap = make(map[string]string)
o.tenantProxyMap = make(map[string]*httputil.ReverseProxy)
o.PopulateTenantHostMap()
// Rx
http.HandleFunc("/", o.ServeHTTP)
go http.ListenAndServe(":5555", nil)
}
// ServeHTTP is the callback that is called each time a Http Request is received.
func (o *Offloader) ServeHTTP(w http.ResponseWriter, req *http.Request) {
incomingUrl := req.URL.RequestURI()
tenant := o.GetTenantFromUrl(incomingUrl)
if proxy, ok := o.tenantProxyMap[tenant]; ok {
proxy.ServeHTTP(w, req)
}
if remoteHostAddr, ok := o.tenantHostMap[tenant]; ok {
remoteUrl, err := url.Parse(fmt.Sprintf("http://%s", remoteHostAddr))
if err != nil {
return
}
proxy := httputil.NewSingleHostReverseProxy(remoteUrl)
o.tenantProxyMap[tenant] = proxy
proxy.ServeHTTP(w, req) // non blocking
} else {
panic("Unknown Tenant")
}
}
When receiving a new HTTP request, I get the tenant from the URL. If this is the first time I am seeing this tenant I create a new ReverseProxy, otherwise I try to use the one I created before and stored in the tenantProxyMap.
When I test this, I get the following error:
2022/04/05 12:31:01 http: proxy error: readfrom tcp ****: http: invalid Read on closed Body
2022/04/05 12:31:01 http: superfluous response.WriteHeader call from net/http/httputil.(*ReverseProxy).defaultErrorHandler (reverseproxy.go:190)
If I create a new Reverse Proxy for each request rather than reusing the same proxy, the error doesn't happen.
I thought the proxy is per host and not per request (as the name suggests), so I am wondering why this error happens?
I know I need to protect the maps from concurrent reads/writes however that is irrelevant at the moment.
Thanks,
The problem is that in the scenario where a previous proxy already existed, you first pass the request on to that - and then still recreate the proxy, and again pass the request. In other words: you are making two proxied requests for each incoming request, when the tentantProxyMap is already populated for that tenant.
The ReverseProxy implementation closes the req.Body, so the second time you pass the request on to the proxy, it attempts reading from an already closed body. You're seeing the http: invalid Read on closed Body error as a result.
What you should try is to return after proxying the request, e.g. by adding a return:
if proxy, ok := o.tenantProxyMap[tenant]; ok {
proxy.ServeHTTP(w, req)
return
}
How to I get the IP address of the client in Deno?
I have created a test server using the standard http library but I can't figure out a way to extract client's IP.
I need that as a security feature for preventing multiple submissions.
In NodeJS/Express there is an ip property of the request object that does the same.
req.ip gives the thing I want in Express but what is it's equivalent in Deno?
My code is:
import { serve } from "https://deno.land/std#0.125.0/http/server.ts";
serve(
(req) => {
console.log(/* the client IP */);
return new Response("hello");
},
{ port: 8080 }
);
Is there any other work-around to prevent multiple access from the same device?
Thanks
To do this in a type-safe way is a little complicated because of the way that serve is typed. First, I'll show you an example of how to do it, then I'll explain the types afterward.
Example
example.ts:
import {
serve,
type ConnInfo,
type Handler,
type ServeInit,
} from 'https://deno.land/std#0.125.0/http/server.ts';
function assertIsNetAddr (addr: Deno.Addr): asserts addr is Deno.NetAddr {
if (!['tcp', 'udp'].includes(addr.transport)) {
throw new Error('Not a network address');
}
}
function getRemoteAddress (connInfo: ConnInfo): Deno.NetAddr {
assertIsNetAddr(connInfo.remoteAddr);
return connInfo.remoteAddr;
}
const handler: Handler = (request, connInfo) => {
const {hostname, port} = getRemoteAddress(connInfo);
const message = `You connected from the following address: ${hostname}`;
return new Response(message);
};
const init: ServeInit = {port: 8080};
serve(handler, init);
console.log(`Listening on port ${init.port}...\nUse ctrl+c to stop`);
Types
Looking at the documentation for the serve function, you can see that it accepts two parameters: a callback of type Handler, and some options of type ServeInit:
async function serve(handler: Handler, options?: ServeInit): Promise<void>;
The Handler callback accepts two parameters: a Request, and an object of type ConnInfo:
type Handler = (request: Request, connInfo: ConnInfo) => Response | Promise<Response>;
ConnInfo looks like this:
interface ConnInfo {
readonly localAddr: Deno.Addr;
readonly remoteAddr: Deno.Addr;
}
The part that should have the remote IP address (technically, it's the remote hostname, but it's very likely to be an IP address unless you have configured custom DNS settings in your server environment) is the object at connInfo.remoteAddr, which (you can see above) is of type Deno.Addr, which looks like this:
// in the Deno namespace
type Addr = NetAddr | UnixAddr;
This is where it becomes complicated. Deno.Addr is a discriminated union of Deno.NetAddr and Deno.UnixAddr (which means that it could be either one), and the property transport is used to discriminate between the two.
// in the Deno namespace
interface NetAddr {
hostname: string;
port: number;
transport: "tcp" | "udp";
}
interface UnixAddr {
path: string;
transport: "unix" | "unixpacket";
}
A net address has hostname property (the value of which would be the IP address) and a port property, while a unix address has a path property.
The listener which is created internally to support the server is actually only listening on TCP, so I think it's safe to assume that the remote address will be a net address. However, because the type signature of the Handler callback parameter in the serve function doesn't make this explicit (although it should!), TypeScript doesn't know that.
So, it's left up to you as the programmer to make sure that the address is actually a net address before you can access properties that would be on a net address (instead of a unix address) in a type-safe way. That's where the type assertion function assertIsNetAddr comes into play. (A type assertion performs a runtime test which results in a "guarantee" of a condition to the compiler: by throwing an exception if the condition can't be guaranteed.) Because you as the programmer already know more than the TypeScript compiler (that the address is on TCP and will be a net address), you can assert that the address is indeed a net address. Then the compiler will allow you use the address as a net address.
If you want to do something besides throwing an Error in the case that the address is not a net address: instead of an assertion function, you can use a type predicate as a condition in your code.
Here is a link to the TypeScript Playground where I've created a playground with the types used in my example, so you can explore/experiment.
Finally, (this is not type-safe) if you just want to use the value without the checks (because you've done your research and are confident that you will never handle a non-TCP connection, you can simply use a type assertion:
const handler: Handler = (request, connInfo) => {
const {hostname, port} = connInfo.remoteAddr as Deno.NetAddr;
const message = `You connected from the following address: ${hostname}`;
return new Response(message);
};
When starting an http server using HTTP.serve there is apparently no way to reload the code that is actually handling the HTTP request.
In the example below I would like to have the modifications in my_httphandler taken into account without having to restart the server.
For the moment I need to stop the server from the REPL by pressing CTRL+C twice and then run the script again.
Is there a workaround ?
module MyModule
using HTTP
using Mux
using JSON
using Sockets
function my_httphandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world")
end
const MY_ROUTER = HTTP.Router()
HTTP.#register(MY_ROUTER, "GET", "/*", my_httphandler)
HTTP.serve(MY_ROUTER, Sockets.localhost, 8081)
end
I'm not sure whether Mux caches handlers. As long as it does not, this should work:
module MyModule
using HTTP
using Mux
using JSON
using Sockets
function my_httphandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world")
end
const functionref = Any[my_httphandler]
const MY_ROUTER = HTTP.Router()
HTTP.#register(MY_ROUTER, "GET", "/*", functionref[1])
HTTP.serve(MY_ROUTER, Sockets.localhost, 8081)
end
function newhandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world 2")
end
MyModule.functionref[1] = newhandler
Revise.jl lets you automatically update code in a live Julia session. You may be especially interested in entr; see Revise's documentation for details.
When using HTTP.jl: just add #async before HTTP.serve
module MyModule
using HTTP
using Sockets
function my_httphandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world")
end
const MY_ROUTER = HTTP.Router()
HTTP.#register(MY_ROUTER, "GET", "/*", my_httphandler)
#async HTTP.serve(MY_ROUTER, Sockets.localhost, 8081)
end # module
When using Mux.jl: nothing to do, the server is started in the background
using Mux
function sayhellotome(name)
return("hello " * name * "!!!")
end
#app test = (
Mux.defaults,
route("/sayhello/:user", req -> begin
sayhellotome(req[:params][:user])
end),
Mux.notfound())
Mux.serve(test, 8082)
I've added a ticket #587 to HTTP.jl project for developer workflow support. I'm not sure this is your use case or not.
# hello.jl -- an example showing how Revise.jl works with HTTP.jl
# julia> using Revise; includet("hello.jl"); serve();
using HTTP
using Sockets
homepage(req::HTTP.Request) =
HTTP.Response(200, "<html><body>Hello World!</body></html>")
const ROUTER = HTTP.Router()
HTTP.#register(ROUTER, "GET", "/", homepage)
serve() = HTTP.listen(request -> begin
Revise.revise()
Base.invokelatest(HTTP.handle, ROUTER, request)
end, Sockets.localhost, 8080, verbose=true)
Alternatively, you could have a test/serve.jl file, that assumes MyModule with a top-level HTTP.jl router is called ROUTER. You'll need to remove the call to serve in your main module.
#!/usr/bin/env julia
using HTTP
using Sockets
using Revise
using MyModule: ROUTER
HTTP.listen(request -> begin
Revise.revise()
Base.invokelatest(HTTP.handle, ROUTER, request)
end, Sockets.localhost, 8080, verbose=true)
A more robust solution would catch errors; however, I had challenges getting this to work and reported my experience at #541 in Revise.jl.