I am doing a reverse proxy thing which just forwards your request on another server but I am unable to set the body.
// post request from endpoint /forwarder.
// Called this using Router.post('/forwarder', ReverseProxy())
func ReverseProxy() gin.HandlerFunc {
endpoint := "http://localhost:3001"
remote, _ := url.Parse(endpoint)
proxy := httputil.NewSingleHostReverseProxy(remote)
return func(c *gin.Context) {
body, _ := ioutil.ReadAll(c.Request.Body)
body_string := string(body)
fmt.Printf("this is data from rqeuest\n%v\n", c.Request.Body)
//Define the director func
proxy.Director = func(req *http.Request) {
req.Header = c.Request.Header
req.Body = ioutil.NopCloser(strings.NewReader(body_string))
req.ContentLength = int64(len(body_string))
fmt.Printf("this is data different \n%v\n", req.Body)
temp, _ := ioutil.ReadAll(req.Body)
fmt.Printf("this is data from req.body\n%v\n", string(temp))
req.Host = remote.Host
req.URL.Scheme = remote.Scheme
req.URL.Host = remote.Host
req.URL.Path = c.Param("proxyPath")
}
proxy.ServeHTTP(c.Writer, c.Request)
}
}
This results in
http: proxy error: http: ContentLength=43 with Body length 0
Removing the req.ContentLength = int64(len(body_string)) forwards the request to the server running at localhost:3001 but the body is undefined.
How do I set the body?
Related
I'm starting to work with fasthttp in Golang and I can't figure out how to send key:value format. In default net/http I did it via url.values. I would appreciate it if you could help me with some sample code!
Image from Burp Suite (How it must look like)
var client *fasthttp.Client
var headerContentTypeJson = []byte("application/json")
type loginData struct {
login string
pass string
}
func main() {
readTimeout, _ := time.ParseDuration("500ms")
writeTimeout, _ := time.ParseDuration("500ms")
maxIdleConnDuration, _ := time.ParseDuration("1h")
client = &fasthttp.Client{
ReadTimeout: readTimeout,
WriteTimeout: writeTimeout,
MaxIdleConnDuration: maxIdleConnDuration,
NoDefaultUserAgentHeader: true, //
DisableHeaderNamesNormalizing: true, //
DisablePathNormalizing: true,
Dial: (&fasthttp.TCPDialer{
Concurrency: 4096,
DNSCacheDuration: time.Hour,
}).Dial,
}
reqTimeout := time.Duration(100) * time.Millisecond
reqData := &loginData{
login: "login",
pass: "password",
}
reqDataByte, _ := json.Marshal(reqData)
req := fasthttp.AcquireRequest()
req.SetRequestURI("https://oskelly.ru/api/v2/account/rawauth")
req.Header.SetMethod(fasthttp.MethodPost)
req.Header.SetContentTypeBytes(headerContentTypeJson)
req.SetBodyRaw(reqDataByte)
resp := fasthttp.AcquireResponse()
err := client.DoTimeout(req, resp, reqTimeout)
fasthttp.ReleaseRequest(req)
if err == nil {
statusCode := resp.StatusCode()
respBody := resp.Body()
fmt.Printf("DEEBUG Response: %s\n", respBody)
if statusCode == http.StatusOK {
respData := &loginData{}
err := json.Unmarshal(respBody, respData)
if err == io.EOF || err == nil {
fmt.Printf("DEBUG Parsed data Response %v\n")
} else {
fmt.Printf("ERR invalid HTTP response code: %d\n", statusCode)
}
}
fasthttp.ReleaseResponse(resp)
}}
enter image description here
Tried to figure out how to integrate url.values into fasthttp
For request Body parameters:
req.Header.SetContentType("application/x-www-form-urlencoded; charset=UTF-8")
Form the request body in key=val string and not json. Can do a simple Stringer impl or req.PostArgs().Add(key, val)
req.PostArgs().Add("login", reqData.Login)
req.PostArgs().Add("pass", reqData.Pass)
Output request
POST / HTTP/1.1
Host: ....
Content-Length: 25
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
login=login&pass=password
I am trying to send request from Go/wasm with Go's net/http package (I am not sure if I should use javascript's fetch function from wasm) . I cannot reach all of the response Headers and Cookies from Go/WASM even though I can see properly all of the headers and cookies on the browser (on network tab of browser and also i can see all of the headers with curl request) . When I try to print all of the headers i can only see 2 headers on the console.Those are "Content-Length" and "Content-Type" . Does anybody knows what is the reason of this ?
Here is example code of server side:
import "github.com/gorilla/sessions"
var store = sessions.NewCookieStore([]byte("super-secret-key-4"))
func (a *App) TestHandler(w http.ResponseWriter, r *http.Request) {
cookieSession, _ := store.Get(r, "session")
cookieSession.Values["test"] = "test"
cookieSession.Save(r, w)
w.Header().Set("Test", "test")
io.WriteString(w, `{"test":"test"}`)
return
}
Client Side:
func TestRequest(userName string) {
type Payload struct {
Name string `json:"name"`
}
payload := Payload{
Name: userName,
}
payloadBytes, _ := json.Marshal(payload)
body := bytes.NewReader(payloadBytes)
req, _:= http.NewRequest("POST","localhost:8080/Test", body)
req.Header.Set("Content-Type", "application/json")
resp, _:= http.DefaultClient.Do(req)
//a, _ := ioutil.ReadAll(resp.Body)
//bodyString := string(a)
for name, values := range resp.Header {
for _, value := range values {
log.Println(name, value)
}
}
for _, cookie := range resp.Cookies() {
log.Println(cookie.Name)
}
defer resp.Body.Close()
}
This is what i get on the browser console:
wasm_exec.js:51 2021/08/04 21:08:48 Content-Length 274
wasm_exec.js:51 2021/08/04 21:08:48 Content-Type text/plain; charset=utf-8
I finally see that i need to add a special header to response for use my custom headers on client side.If i add this line to my middleware i can see my custom headers on client side.
w.Header().Set("Access-Control-Expose-Headers",
"MyHeader,Myheader2")
I would like to implement an HTTP request and response system of this for
client ------> A ----------> B --------> (request) HTTPserver
client <------ A <---------- B <-------- (response) HTTPserver
client send HTTP request with any HTTP method (POST,PUT, etc) to A
A then reads the request body encrypt it and then forward it to B
3 B then decrypt the reads the HTTP request body
B then with the received decrypt body as payload prepare an HTTP request and and send to the HTTP server.
5 The HTTP server then respond to B.
B then encrypt the response from the HTTP server and then forward to A.
A also decrypt the response from B and then and send the response back to the client.
I have implements the following base on earlier suggestions.
ProxyA:
const (
ServerB = "<address of B>"
Port = "<proxy A port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
// read request body from client
bodybytes, _ := ioutil.ReadAll(req.Body)
defer req.Body.Close()
// encrypt req.Body
object, _ := enc.Encrypt(bodybytes)
// serialize object
serialized, _ := object.CompactSerialize()
// prepare forwarding message
msg := message{reformatedData: serialized}
// encode message
msgbytes, _ := json.Marshal(&msg)
req.ContentLength = int64(len(msgbytes))
req.Body = ioutil.NopCloser(bytes.NewBuffer(msgbytes))
// How do I read the response data from proxy server B and then send
// response to the client
....
url, _ := url.Parse(ServerB)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
For proxy B:
const (
Server = "<address of server>"
Port = "<proxy B port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
var msg message
HTTPServerurl := http://xxxxx
// read request body
bodybytes, _ := ioutil.ReadAll(req.Body)
req.ContentLength = int64(len(bodybytes))
req.Body = ioutil.NopCloser(bytes.NewBuffer(bodybytes))
// decode message
err = json.Unmarshal(bodybytes, &msg)
// decrypt message
object, _ := jose.ParseEncrypted(msg)
decrypted, _ := object.Decrypt("phrasetodecryptmessage")
//send HTTP request to HTTP server
resp, _ := HandlemessageToHTTPServer(decrypted, "POST", HTTPServerurl)
//read response body
RespBody, _ := ioutil.ReadAll(resp.Body)
// encrypt response body
object, _ = enc.Encrypt(producerRespBody)
serialized, _ := object.CompactSerialize()
// prepare response JSON message
resmsg := resmessage {resmessage: serialized}
// marshall response message
respmsgbytes, _ := json.Marshal(&resmsg)
// How do I write the "respmsgbytes" to proxyServHTTP "res" back to proxy A
url, _ := url.Parse(Server)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
My question is
How do I write the "respmsgbytes" to proxyServHTTP "res" in proxy B back to proxy A ?
How do I read the response data from proxy server B and then send
response to the client?
Any help? I have left error checking to make the code short.
You can use httputil
You can do something like following.
For proxy A:
const (
ServerB = "<address of B>"
Port = "<proxy A port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
// Encrypt Request here
// ...
url, _ := url.Parse(ServerB)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
For proxy B:
const (
Server = "<address of server>"
Port = "<proxy B port>"
)
func main() {
// start server
http.HandleFunc("/", proxyPass)
log.Fatal(http.ListenAndServe(":" + Port, nil))
}
func proxyPass(res http.ResponseWriter, req *http.Request) {
// Decrypt Request here
// ...
url, _ := url.Parse(Server)
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(res, req)
}
EDIT:
To handle the request body at each proxy, you can have a look at this. Alternatively, I think there should be no harm in construction of new req based on current req as following:
func proxyPass(res http.ResponseWriter, req *http.Request) {
body, _ := ioutil.ReadAll(req.Body)
data := string(body)
// process data here
req, _ = http.NewRequest(req.Method, req.URL.String(), strings.NewReader(data))
u, _ := url.Parse(Server)
proxy := httputil.NewSingleHostReverseProxy(u)
proxy.ServeHTTP(res, req)
}
This can be done at both proxies.
EDIT:
The proxy response can be updated using ReverseProxy.ModifyResponse.
You can use it like this:
func proxyPass(res http.ResponseWriter, req *http.Request) {
....
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ModifyResponse = func(response *http.Response) error {
// Read and update the response here
// The response here is response from server (proxy B if this is at proxy A)
// It is a pointer, so can be modified to update in place
// It will not be called if Proxy B is unreachable
}
proxy.ServeHTTP(res, req)
}
I'm trying to do a load balancer to study some go packages.
I want to handle errors when the request timeout or give 404 error but can't find how to do that.
func main() {
// start server
http.HandleFunc("/", handleRequestAndRedirect)
if err := http.ListenAndServe(getListenAddress(), nil); err != nil {
panic(err)
}
}
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
ur, _ := url.Parse("https://www.instagram.com/")
proxy := httputil.NewSingleHostReverseProxy(ur)
// Update the headers to allow for SSL redirection
req.URL.Host = ur.Host
req.URL.Scheme = ur.Scheme
req.Header.Set("X-Forwarded-Host", req.Header.Get("Host"))
req.Host = ur.Host
req.Header.Set("Key", "Teste")
proxy.ServeHTTP(res, req)
}
As I ended up here looking for a way to handle 404 errors from the proxied host, I would like to complement the accepted answer, if it may be of any help for people landing on this page.
As stated in the official documentation (https://golang.org/pkg/net/http/httputil/#ReverseProxy):
ModifyResponse is an optional function that modifies the Response from the backend. It is called if the backend returns a response at all, with any HTTP status code.
If the backend is unreachable, the optional ErrorHandler is called without any call to ModifyResponse. If ModifyResponse returns an error, ErrorHandler is called with its error value. If ErrorHandler is nil, its default implementation is used.
So if you want to catch not only "real" errors (host not reachable) but also error response codes from the server (404, 500...) you should use ModifyResponse to check the response status code and return an error, which will be then catched by your ErrorHandler function. The accepted answer example becomes:
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
ur, _ := url.Parse("https://www.instagram.com/")
proxy := httputil.NewSingleHostReverseProxy(ur)
// Update the headers to allow for SSL redirection
req.URL.Host = ur.Host
req.URL.Scheme = ur.Scheme
req.Header.Set("X-Forwarded-Host", req.Header.Get("Host"))
req.Host = ur.Host
req.Header.Set("Key", "Teste")
proxy.ErrorHandler = ErrHandle
proxy.ModifyResponse = ModifyResponse
proxy.ServeHTTP(res, req)
}
func ModifyResponse(res *http.Response) error {
if res.StatusCode == 404 {
return errors.New("404 error from the host")
}
return nil
}
func ErrHandle(res http.ResponseWriter, req *http.Request, err error) {
fmt.Println(err)
}
use proxy.ErrorHandler
ErrorHandler func(http.ResponseWriter, *http.Request, error)
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
ur, _ := url.Parse("https://www.instagram.com/")
proxy := httputil.NewSingleHostReverseProxy(ur)
// Update the headers to allow for SSL redirection
req.URL.Host = ur.Host
req.URL.Scheme = ur.Scheme
req.Header.Set("X-Forwarded-Host", req.Header.Get("Host"))
req.Host = ur.Host
req.Header.Set("Key", "Teste")
proxy.ErrorHandler = ErrHandle
proxy.ServeHTTP(res, req)
}
func ErrHandle(res http.ResponseWriter, req *http.Request, err error) {
fmt.Println(err)
}
I need to use a proxy with auth using PostForm method.
If I use something like (simplified):
request, err := http.NewRequest("GET", url.String(), nil)
response, err := client.Do(request)
I can with ease do request.Header.Add("Proxy-Authorization", basicAuth) and it works fine.
But now, I am editing third-party package, and I try to add proxy to the existing code:
proxyStr := "http://proxy.com:8080"
proxyURL, _ := url.Parse(proxyStr)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
bot.Client = &http.Client{
Transport: transport,
}
resp, err := bot.Client.PostForm(method, params)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
resp.Header.Add("Proxy-Authorization", basicAuth)
It does not work, and it fails, to my mind, at string resp.Header.Add("Proxy-Authorization", basicAuth).
Proxy without auth works fine, in this example.
Does anybody know, can I use proxy with auth in this case?
You can create the client once by using the following code. Then substitute your HTTP client in the third-party package.
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(&url.URL{
Scheme: "http",
User: url.UserPassword("username", "password"),
Host: "146.137.9.45:65233",
}),
},
}
or you can parse the URL as well
url, _ := url.Parse("http://username:password#146.137.9.45:65233")
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(url),
}
}
You are trying to add a header to a response, which isn't what you send to the server but what you receive. You have to add headers and data to the request, which you have to assemble first and then execute it like this:
data := url.Values{} // the form data
data.Add("foo-key", "some data")
req, err := http.NewRequest("POST","https://yoururl", strings.NewReader(data.Encode()))
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
req.Header.Add("Proxy-Authorization", basicAuth)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := bot.Client.Do(req)
Then you just use the response (resp)
Thanks to all!
I found such a solution (may be it would be useful to someone):
// Uncomment to use proxy with auth
/*
proxyStr := "http://proxy.com:3128"
proxyURL, _ := url.Parse(proxyStr)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
hdr := http.Header{}
hdr.Add("Proxy-Authorization", basicAuth)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
ProxyConnectHeader: hdr,
}
bot.Client = &http.Client{
Transport: transport,
}
*/
resp, err := bot.Client.PostForm(method, params)