How to implement HTTP forwarding Proxy in Go - http

I would like to implement an HTTP request and response system of this for
client ------> A ----------> B --------> (request) HTTPserver
client <------ A <---------- B <-------- (response) HTTPserver
client send HTTP request with any HTTP method (POST,PUT, etc) to A
A then reads the request body encrypt it and then forward it to B
3 B then decrypt the reads the HTTP request body
B then with the received decrypt body as payload prepare an HTTP request and and send to the HTTP server.
5 The HTTP server then respond to B.
B then encrypt the response from the HTTP server and then forward to A.
A also decrypt the response from B and then and send the response back to the client.
I have implements the following base on earlier suggestions.
ProxyA:
const (
ServerB = "<address of B>"
Port = "<proxy A port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
// read request body from client
bodybytes, _ := ioutil.ReadAll(req.Body)
defer req.Body.Close()
// encrypt req.Body
object, _ := enc.Encrypt(bodybytes)
// serialize object
serialized, _ := object.CompactSerialize()
// prepare forwarding message
msg := message{reformatedData: serialized}
// encode message
msgbytes, _ := json.Marshal(&msg)
req.ContentLength = int64(len(msgbytes))
req.Body = ioutil.NopCloser(bytes.NewBuffer(msgbytes))
// How do I read the response data from proxy server B and then send
// response to the client
....
url, _ := url.Parse(ServerB)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
For proxy B:
const (
Server = "<address of server>"
Port = "<proxy B port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
var msg message
HTTPServerurl := http://xxxxx
// read request body
bodybytes, _ := ioutil.ReadAll(req.Body)
req.ContentLength = int64(len(bodybytes))
req.Body = ioutil.NopCloser(bytes.NewBuffer(bodybytes))
// decode message
err = json.Unmarshal(bodybytes, &msg)
// decrypt message
object, _ := jose.ParseEncrypted(msg)
decrypted, _ := object.Decrypt("phrasetodecryptmessage")
//send HTTP request to HTTP server
resp, _ := HandlemessageToHTTPServer(decrypted, "POST", HTTPServerurl)
//read response body
RespBody, _ := ioutil.ReadAll(resp.Body)
// encrypt response body
object, _ = enc.Encrypt(producerRespBody)
serialized, _ := object.CompactSerialize()
// prepare response JSON message
resmsg := resmessage {resmessage: serialized}
// marshall response message
respmsgbytes, _ := json.Marshal(&resmsg)
// How do I write the "respmsgbytes" to proxyServHTTP "res" back to proxy A
url, _ := url.Parse(Server)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
My question is
How do I write the "respmsgbytes" to proxyServHTTP "res" in proxy B back to proxy A ?
How do I read the response data from proxy server B and then send
response to the client?
Any help? I have left error checking to make the code short.

You can use httputil
You can do something like following.
For proxy A:
const (
ServerB = "<address of B>"
Port = "<proxy A port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
// Encrypt Request here
// ...
url, _ := url.Parse(ServerB)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
For proxy B:
const (
Server = "<address of server>"
Port = "<proxy B port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
// Decrypt Request here
// ...
url, _ := url.Parse(Server)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
EDIT:
To handle the request body at each proxy, you can have a look at this. Alternatively, I think there should be no harm in construction of new req based on current req as following:
func proxyPass(res http.ResponseWriter, req *http.Request) {
body, _ := ioutil.ReadAll(req.Body)
data := string(body)
// process data here
req, _ = http.NewRequest(req.Method, req.URL.String(), strings.NewReader(data))
u, _ := url.Parse(Server)
proxy := httputil.NewSingleHostReverseProxy(u)
proxy.ServeHTTP(res, req)
}
This can be done at both proxies.
EDIT:
The proxy response can be updated using ReverseProxy.ModifyResponse.
You can use it like this:
func proxyPass(res http.ResponseWriter, req *http.Request) {
....
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ModifyResponse = func(response *http.Response) error {
// Read and update the response here
// The response here is response from server (proxy B if this is at proxy A)
// It is a pointer, so can be modified to update in place
// It will not be called if Proxy B is unreachable
}
proxy.ServeHTTP(res, req)
}

Related

How to make a HTTP request from server to client using grpc in golang

Problem Statement
I have a client (which dials to the server) and server (that listens for incoming requests) written in golang and with the RPC calls defined. I am trying to initiate an HTTP request on the server side which would in turn execute the RPC call for streaming and send a JSON response back to the user
Challenge
I was able to handle both grpc and HTTP requests on different ports but having issues with passing parameters from the HTTP request onto the RPC call on the server side
Server Code
log.Println("Listening for connections from client ........")
lis, err := net.Listen("tcp", ":9000")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := testApi.Server{}
grpcServer := grpc.NewServer()
testApi.RegisterTestApiServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %s", err)
}
func main() {
go runGrpc()
log.Printf("*------ Waiting for requests from users ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":8080", router))
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
test_id, _ := strconv.Atoi(vars["test_id"])
log.Println("Test id request from user : ", test_id)
func (s * Server) ConnectAndStream(channelStream TestApiService_ConnectAndStreamServer) error {
// Question: This Id has to come from http request above- test_id
var id int32 = 1234566
// id := a.ConnectAndExchange
log.Println("Id from sam user ", id)
// var id int32 = 1234566
for i := 1; i <= 2; i++ {
id += 1
log.Println("Speed Server is sending data : ", id)
channelStream.Send(&Input{Id: id})
}
for i := 1; i <= 2; i++ {
log.Println("now time to receive")
client_response, err := channelStream.Recv()
log.Println("Response from samd client : ", client_response.Id)
if err != nil {
log.Println("Error while receiving from samd : ", err)
}
}
return nil
}
I am stuck with being able to pass the test_id from the curl request to the RPC call as above. Any input is greatly appreciated
Note
Client - Dials in and connects to the server and starts receiving and sending data (bi-directional streaming)
Both the Http and GRPC client are part of the same server application. So why call the RPC method from the Http handler? The Http handler should have access to the same backend functionality.
Your question is slightly unclear but if you are trying to have your client establish a GRPC connection to the server via the HTTP handler this will not work. The GRPC connection established in this situation is between the server and its self.
Edit - thanks for the clarification. Now I understand better the flow that you are trying to achieve. Your http handler method can make the outgoing grpc call to the server and return the response back via the http.ResponseWriter
For simplicity I have used the hello world example on https://github.com/grpc/grpc-go/tree/master/examples/helloworld
Running the code sample below and hitting http://localhost:1000/exchangeId/Test will show the output
Starting
*------ Waiting for http requests from users on port 1000 ------*
server listening at 127.0.0.1:1001
Test id request from user : Test
Server Received: Test
Greeting: Hello Test
Code sample:
import (
"context"
"log"
"net"
"net/http"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
"github.com/gorilla/mux"
)
var (
grpcserver = "localhost:1001"
)
func main() {
log.Print("Starting")
go StartGrpcServer()
log.Printf("*------ Waiting for http requests from users on port 1000 ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":1000", router))
}
type server struct {
pb.UnimplementedGreeterServer
}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Server Received: %v", in.GetName())
return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}
func StartGrpcServer() {
lis, err := net.Listen("tcp", grpcserver)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
log.Printf("server listening at %v", lis.Addr())
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
test_id := vars["test_id"]
log.Println("Test id request from user : ", test_id)
// Set up a connection to the server.
conn, err := grpc.Dial(grpcserver, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
// Contact the server and print out its response.
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
resp, err := c.SayHello(ctx, &pb.HelloRequest{Name: test_id})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", resp.GetMessage())
w.Write([]byte(resp.GetMessage()))
}

Set request body in httputil reverse proxy

I am doing a reverse proxy thing which just forwards your request on another server but I am unable to set the body.
// post request from endpoint /forwarder.
// Called this using Router.post('/forwarder', ReverseProxy())
func ReverseProxy() gin.HandlerFunc {
endpoint := "http://localhost:3001"
remote, _ := url.Parse(endpoint)
proxy := httputil.NewSingleHostReverseProxy(remote)
return func(c *gin.Context) {
body, _ := ioutil.ReadAll(c.Request.Body)
body_string := string(body)
fmt.Printf("this is data from rqeuest\n%v\n", c.Request.Body)
//Define the director func
proxy.Director = func(req *http.Request) {
req.Header = c.Request.Header
req.Body = ioutil.NopCloser(strings.NewReader(body_string))
req.ContentLength = int64(len(body_string))
fmt.Printf("this is data different \n%v\n", req.Body)
temp, _ := ioutil.ReadAll(req.Body)
fmt.Printf("this is data from req.body\n%v\n", string(temp))
req.Host = remote.Host
req.URL.Scheme = remote.Scheme
req.URL.Host = remote.Host
req.URL.Path = c.Param("proxyPath")
}
proxy.ServeHTTP(c.Writer, c.Request)
}
}
This results in
http: proxy error: http: ContentLength=43 with Body length 0
Removing the req.ContentLength = int64(len(body_string)) forwards the request to the server running at localhost:3001 but the body is undefined.
How do I set the body?

http forward with domain blocking

I'm trying to implement a http forwarding server that supports domain blocking. I've tried
go io.Copy(dst, src)
go io.Copy(src, dst)
and it works like a charm on tcp forwarding. Then I've tried to do request line parsing with something similar to
go func(){
reader := io.TeeReader(src, dst)
textReader := textproto.NewReader(bufio.NewReader(reader))
requestLine, _ = textReader.ReadLine()
// ...
ioutil.ReadAll(reader)
}
It works fine, but I was getting worried about bad performance(with ioutil.ReadAll). So I've written the code below.
func (f *Forwarder) handle(src, dst net.Conn) {
defer dst.Close()
defer src.Close()
done := make(chan struct{})
go func() {
textReader := bufio.NewReader(src)
requestLine, _ = textReader.ReadString('\n')
// parse request line and apply domain blocking
dst.Write([]byte(requestLine))
io.Copy(dst, src)
done <- struct{}{}
}()
go func() {
textReader := bufio.NewReader(dst)
s.statusLine, _ = textReader.ReadString('\n')
src.Write([]byte(s.statusLine))
io.Copy(src, dst)
done <- struct{}{}
}()
<-done
<-done
}
Unfortunately, it doesn't work at all. Requests get to print out, but not for responses. I've stuck here and don't know what's wrong.
TCP forwarding is to realize that the tunnel proxy does not need to parse data. The reverse proxy can use the standard library.
The tunnel proxy is implemented to separate the http and https protocols. The client generally uses the tunnel to send https and sends the Connect method. Sending http is the Get method. For the https request service, only dail creates the connection tcp conversion, and the http request is implemented using a reverse proxy.
func(w http.ResponseWriter, r *http.Request) {
// check url host
if r.URL.Host != "" {
if r.Method == eudore.MethodConnect {
// tunnel proxy
conn, err := net.Dial("tcp", r.URL.Host)
if err != nil {
w.WriteHeader(502)
return
}
client, _, err := w.Hijack()
if err != nil {
w.WriteHeader(502)
conn.Close()
return
}
client.Write([]byte("HTTP/1.0 200 OK\r\n\r\n"))
go func() {
io.Copy(client, conn)
client.Close()
conn.Close()
}()
go func() {
io.Copy(conn, client)
client.Close()
conn.Close()
}()
} else {
// reverse proxy
httputil.NewSingleHostReverseProxy(r.URL).ServeHTTP(w, r)
}
}
}
Implementing a reverse proxy will parse the client request, and the proxy will send the request to the target server.
Reverse proxy conversion request, not tested :
func(w http.ResponseWriter, r *http.Request) {
// set host
r.URL.Scheme = "http"
r.URL.Path = "example.com"
// send
resp,err := http.DefaultClient.Do(r)
if err != nil {
w.WriteHeader(502)
return
}
// write respsonse
defer resp.Body.Close()
w.WriteHeader(resp.StatusCode)
h := w.Header()
for k,v := range resp.Header {
h[k]=v
}
io.Copy(w, resp.Body)
}
However, the direct forwarding request does not process the hop-to-hop header. The hop-to-hop header is clearly stated in the rfc. The hop-to-hop header is the transmission information between two connections. For example, the client to the proxy and the proxy to the server are two. And the client to the server is end-to-end.
Please use the standard library directly for the reverse proxy, it has already handled the hop-to-hop header and Upgrade for you.
exmample NewSingleHostReverseProxy with filter:
package main
import (
"net/http"
"strings"
"net/http/httputil"
"net/url"
)
func main() {
addr, _ := url.Parse("http://localhost:8088")
proxy := httputil.NewSingleHostReverseProxy(addr)
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/api/") {
proxy.ServeHTTP(w, r)
} else {
w.WriteHeader(404)
}
})
// Listen Server
}

How to handle error using NewSingleHostReverseProxy

I'm trying to do a load balancer to study some go packages.
I want to handle errors when the request timeout or give 404 error but can't find how to do that.
func main() {
// start server
http.HandleFunc("/", handleRequestAndRedirect)
if err := http.ListenAndServe(getListenAddress(), nil); err != nil {
panic(err)
}
}
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
ur, _ := url.Parse("https://www.instagram.com/")
proxy := httputil.NewSingleHostReverseProxy(ur)
// Update the headers to allow for SSL redirection
req.URL.Host = ur.Host
req.URL.Scheme = ur.Scheme
req.Header.Set("X-Forwarded-Host", req.Header.Get("Host"))
req.Host = ur.Host
req.Header.Set("Key", "Teste")
proxy.ServeHTTP(res, req)
}
As I ended up here looking for a way to handle 404 errors from the proxied host, I would like to complement the accepted answer, if it may be of any help for people landing on this page.
As stated in the official documentation (https://golang.org/pkg/net/http/httputil/#ReverseProxy):
ModifyResponse is an optional function that modifies the Response from the backend. It is called if the backend returns a response at all, with any HTTP status code.
If the backend is unreachable, the optional ErrorHandler is called without any call to ModifyResponse. If ModifyResponse returns an error, ErrorHandler is called with its error value. If ErrorHandler is nil, its default implementation is used.
So if you want to catch not only "real" errors (host not reachable) but also error response codes from the server (404, 500...) you should use ModifyResponse to check the response status code and return an error, which will be then catched by your ErrorHandler function. The accepted answer example becomes:
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
ur, _ := url.Parse("https://www.instagram.com/")
proxy := httputil.NewSingleHostReverseProxy(ur)
// Update the headers to allow for SSL redirection
req.URL.Host = ur.Host
req.URL.Scheme = ur.Scheme
req.Header.Set("X-Forwarded-Host", req.Header.Get("Host"))
req.Host = ur.Host
req.Header.Set("Key", "Teste")
proxy.ErrorHandler = ErrHandle
proxy.ModifyResponse = ModifyResponse
proxy.ServeHTTP(res, req)
}
func ModifyResponse(res *http.Response) error {
if res.StatusCode == 404 {
return errors.New("404 error from the host")
}
return nil
}
func ErrHandle(res http.ResponseWriter, req *http.Request, err error) {
fmt.Println(err)
}
use proxy.ErrorHandler
ErrorHandler func(http.ResponseWriter, *http.Request, error)
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
ur, _ := url.Parse("https://www.instagram.com/")
proxy := httputil.NewSingleHostReverseProxy(ur)
// Update the headers to allow for SSL redirection
req.URL.Host = ur.Host
req.URL.Scheme = ur.Scheme
req.Header.Set("X-Forwarded-Host", req.Header.Get("Host"))
req.Host = ur.Host
req.Header.Set("Key", "Teste")
proxy.ErrorHandler = ErrHandle
proxy.ServeHTTP(res, req)
}
func ErrHandle(res http.ResponseWriter, req *http.Request, err error) {
fmt.Println(err)
}

How can I make a request with a bearer token in Go

I need to make a GET request to an API with a bearer token in the authorization request. How can I do this in Go? I have the following code, but I haven't had success.
package main
import (
"io/ioutil"
"log"
"net/http"
)
func main() {
url := "https://api.globalcode.com.br/v1/publico/eventos"
resp, err := http.Get(url)
resp.Header.Add("Bearer", "token")
if err != nil {
log.Println("Erro ao realizar request.\n[ERRO] -", err)
}
body, _ := ioutil.ReadAll(resp.Body)
log.Println(string([]byte(body)))
}
For control over HTTP client headers, redirect policy, and other settings, create a Client:
package main
import (
"io/ioutil"
"log"
"net/http"
)
func main() {
url := "https://api.globalcode.com.br/v1/publico/eventos"
// Create a Bearer string by appending string access token
var bearer = "Bearer " + <ACCESS TOKEN HERE>
// Create a new request using http
req, err := http.NewRequest("GET", url, nil)
// add authorization header to the req
req.Header.Add("Authorization", bearer)
// Send req using http Client
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Println("Error on response.\n[ERROR] -", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Println("Error while reading the response bytes:", err)
}
log.Println(string([]byte(body)))
}
The Client's Transport typically has internal state (cached TCP
connections), so Clients should be reused instead of created as
needed. Clients are safe for concurrent use by multiple goroutines.
A Client is higher-level than a RoundTripper (such as Transport) and
additionally handles HTTP details such as cookies and redirects.
For more information on Client and Transport check golang spec for net/http package
I had to add a client.CheckRedirect Function(seen below) in order to pass the Bearer token to the API.
bearer := "Bearer " + token
req, err := http.NewRequest("GET", url, bytes.NewBuffer(nil))
req.Header.Set("Authorization", bearer)
req.Header.Add("Accept", "application/json")
client := &http.Client{}
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
for key, val := range via[0].Header {
req.Header[key] = val
}
return err
}
resp, err := client.Do(req)
if err != nil {
log.Println("Error on response.\n[ERRO] -", err)
} else {
defer resp.Body.Close()
data, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(data))
}
}
I made a super-little-basic library for execute basic request like:
package main
import (
request "github.com/alessiosavi/Requests"
)
func main(){
// Create a key-value list of headers
headers := request.CreateHeaderList(`Accept`, `application/json`, "Authorization", "Bearer "+IAMToken)
resp :=request.SendRequest(`http://your_site.com`, `GET`, headers, nil))
}
Here you can find the request implementation:
https://github.com/alessiosavi/Requests/blob/e7ca66bde738b6224fba2b6f146a8dbee67d3323/Requests.go
Here you can find how i use the library for Bearer Auth and other auth type:
https://github.com/alessiosavi/GoCloudant/blob/a8ad3a7990f04ea728bb327d6faea6af3e5455ca/cloudant.go

Resources