I'm attempting to write a reverse proxy in golang that reuses the proxy but I'm getting a nil pointer for a reason that I can't see.
type Configuration struct {
Proxies map[string]Proxy `mapstructure:"proxies"`
Port int `mapstructure:"port"`
}
type Proxy struct {
Upstream string `mapstructure:"upstream"`
SkipVerify bool `mapstructure:"tls-skip-verify"`
HostProxy *httputil.ReverseProxy
}
Those are the structs that I'm loading. I then iterate through the map to create the ReverseProxy object int the proxy struct and start the service.
for host, proxy := range config.Proxies {
log.Printf("Loading proxies for: %s", host)
log.Printf("Upstream: %s", proxy.Upstream)
log.Printf("SkipVerify: %t", proxy.SkipVerify)
remoteURL, err := url.Parse(proxy.Upstream)
if err != nil {
log.Fatalf("Unable to parse target: %v", err)
}
log.Infof("Remote : %v", remoteURL)
proxy.HostProxy = httputil.NewSingleHostReverseProxy(remoteURL)
proxy.HostProxy.Transport = &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 15 * time.Second,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
TLSClientConfig: &tls.Config{InsecureSkipVerify: proxy.SkipVerify},
}
log.Print(proxy.HostProxy.Transport)
}
log.Info("Starting Up on Port", config.Port)
http.Handle("/", &config)
log.Fatal(http.ListenAndServe(":"+strconv.Itoa(config.Port), &config))
When it gets called I handle it like this:
func (config *Configuration) ServeHTTP(w http.ResponseWriter, r *http.Request) {
host := r.Host
r.Host = r.URL.Host
log.Info(config)
if proxy, ok := config.Proxies[host]; ok {
log.Infof("Processing request for %s", host)
log.Printf("Upstream: %s", proxy.Upstream)
log.Printf("SkipVerify: %t", proxy.SkipVerify)
log.Print(proxy.HostProxy.Transport)
proxy.HostProxy.ServeHTTP(w, r)
} else {
w.Write([]byte("403: Host forbidden " + host))
}
}
But it throws exceptions that the proxy.HostProxy object is nil. All the other data persists and logs correctly. Such as the proxy.Upstream and the proxy.SkipVerify.
Related
I am trying to connect to a Host using Random TLS Fingerprinting. I am using https://github.com/refraction-networking/utls (see my issue i created on https://github.com/refraction-networking/utls/issues/42)
My issue is now, how can i utilize a HTTP or SOCKS5 Proxy while opening that connection?
The Code im using right now is:
package main
import (
"bufio"
"fmt"
"log"
"net"
"net/http"
"net/http/httputil"
"net/url"
"time"
"github.com/refraction-networking/utls"
)
var (
dialTimeout = time.Duration(15) * time.Second
)
var requestHostname = "google.com"
var requestAddr = "172.217.22.110:443"
// this example generates a randomized fingeprint, then re-uses it in a follow-up connection
func HttpGetConsistentRandomized(hostname string, addr , uri string) (*http.Response, error) {
config := tls.Config{ServerName: hostname}
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn := tls.UClient(tcpConn, &config, tls.HelloRandomized)
defer uTlsConn.Close()
err = uTlsConn.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
uTlsConn.Close()
// At this point uTlsConn.ClientHelloID holds a seed that was used to generate
// randomized fingerprint. Now we can establish second connection with same fp
tcpConn2, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn2 := tls.UClient(tcpConn2, &config, uTlsConn.ClientHelloID)
defer uTlsConn2.Close()
err = uTlsConn2.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
return httpGetOverConn(uTlsConn2, uTlsConn2.HandshakeState.ServerHello.AlpnProtocol, uri)
}
func main() {
var response *http.Response
var err error
response, err = HttpGetConsistentRandomized(requestHostname, requestAddr, "/2.0/ssocookie")
if err != nil {
fmt.Printf("#> HttpGetConsistentRandomized() failed: %+v\n", err)
} else {
//fmt.Printf("#> HttpGetConsistentRandomized() response: %+s\n", httputil.DumpResponse(response,true))
dump, err := httputil.DumpResponse(response, true)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+s\n", dump)
}
return
}
func httpGetOverConn(conn net.Conn, alpn string, uri string) (*http.Response, error) {
req := &http.Request{
Method: "GET",
URL: &url.URL{Host: "www." + requestHostname + uri},
Header: make(http.Header),
Host: "www." + requestHostname,
}
req.Proto = "HTTP/1.1"
req.ProtoMajor = 1
req.ProtoMinor = 1
err := req.Write(conn)
if err != nil {
return nil, err
}
return http.ReadResponse(bufio.NewReader(conn), req)
}
As Steffen said, you have to create a proxy dialer first, dial the proxy to create a net.Conn, then use that net.Conn when creating the uTLS Client, before handshaking. For brevity's sake, your custom dialTLS function would look something like:
import (
"crypto/tls"
"net"
"net/url"
"github.com/magisterquis/connectproxy"
"golang.org/x/net/proxy"
utls "github.com/refraction-networking/utls"
)
var proxyString = "http://127.0.0.1:8080"
dialTLS := func(network, addr string, _ *tls.Config) (net.Conn, error) {
proxyURI, _ := url.Parse(proxyString)
switch proxyURI.Scheme {
case "socks5":
proxyDialer, err = proxy.SOCKS5("tcp", proxyString, nil, proxy.Direct)
case "http":
proxyDialer, err = connectproxy.New(proxyURI, proxy.Direct)
}
conn, err := proxyDialer.Dial("tcp", addr)
uconn := utls.UClient(conn, cfg, &utls.HelloRandomizedALPN)
...
}
Two suggestions:
Use the "connectproxy" module referenced above if you intend to tunnel through a HTTP CONNECT proxy.
Make life easier for yourself and take a look at the Meek pluggable transport source for Tor. There's a 'utls.go' module which takes care of everything for you, including setting up either a http or http2 transport depending on the negotiated ALPN protocol. It only supports SOCKS but you could easily adapt it to handle HTTP proxies.
A HTTP proxy and SOCKS proxy work be having some initial proxy specific handshake after the TCP connect. After this handshake is done they provide a normal TCP socket which then can be used for doing the TLS handshake etc. Thus, all you need is to replace your
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
with a proxy specific method to setup the TCP connection. This can be done by using SOCKS5 in x/net/proxy to create the appropriate Dialer or similar using the HTTP CONNECT method is done in connectproxy.
The http.Request struct includes the remote IP and port of the request's sender:
// RemoteAddr allows HTTP servers and other software to record
// the network address that sent the request, usually for
// logging. This field is not filled in by ReadRequest and
// has no defined format. The HTTP server in this package
// sets RemoteAddr to an "IP:port" address before invoking a
// handler.
// This field is ignored by the HTTP client.
**RemoteAddr string**
The http.Response object has no such field.
I would like to know the IP address that responded to the request I sent, even when I sent it to a DNS address.
I thought that net.LookupHost() might be helpful, but 1) it can return multiple IPs for a single host name, and 2) it ignores the hosts file unless cgo is available, which it is not in my case.
Is it possible to retrieve the remote IP address for an http.Response?
Use the net/http/httptrace package and use the GotConnInfo hook to capture the net.Conn and its corresponding Conn.RemoteAddr().
This will give you the address the Transport actually dialled, as opposed to what was resolved in DNSDoneInfo:
package main
import (
"log"
"net/http"
"net/http/httptrace"
)
func main() {
req, err := http.NewRequest("GET", "https://example.com/", nil)
if err != nil {
log.Fatal(err)
}
trace := &httptrace.ClientTrace{
GotConn: func(connInfo httptrace.GotConnInfo) {
log.Printf("resolved to: %s", connInfo.Conn.RemoteAddr())
},
}
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
client := &http.Client{}
_, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
}
Outputs:
~ go run ip.go
2017/02/18 19:38:11 resolved to: 104.16.xx.xxx:443
Another solution I came up with was the hook the DialContext function in the http client transport. This is a specific solution that lets you modify the http.Client instead of the request which may be useful.
We first create a function that returns a hooked dial context
func remoteAddressDialHook(remoteAddressPtr *net.Addr) func(ctx context.Context, network string, address string) (net.Conn, error) {
hookedDialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
originalDialer := &net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
conn, err := originalDialer.DialContext(ctx, network, address)
if err != nil {
return nil, err
}
// conn was successfully created
*remoteAddressPtr = conn.RemoteAddr()
return conn, err
}
return hookedDialContext
}
We can then use this function to create a DialContext that writes to an outparameter
var remoteAddr net.Addr
customTransport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: remoteAddressDialHook(&remoteAddr),
ForceAttemptHTTP2: true,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
customHttpClient := http.Client{
Transport: customTransport,
}
// do what you normally would with a http client, it will then set the remoteAddr to be the remote address
fmt.Println(remoteAddr.String())
I have a client machine with multiple NICs, how do I bind an http.Client in Go to a certain NIC or to a certain SRC IP Address?
Say you have some very basic http client code that looks like:
package main
import (
"net/http"
)
func main() {
webclient := &http.Client{}
req, _ := http.NewRequest("GET", "http://www.google.com", nil)
httpResponse, _ := webclient.Do(req)
defer httpResponse.Body.Close()
}
Is there a way to bind to a certain NIC or IP?
Similar to this question, you need to set the http.Client.Transport field. Setting it to an instance of net.Transport allows you to specify which net.Dialer you want to use. net.Dialer then allows you to specify the local address to make connections from.
Example:
localAddr, err := net.ResolveIPAddr("ip", "<my local address>")
if err != nil {
panic(err)
}
// You also need to do this to make it work and not give you a
// "mismatched local address type ip"
// This will make the ResolveIPAddr a TCPAddr without needing to
// say what SRC port number to use.
localTCPAddr := net.TCPAddr{
IP: localAddr.IP,
}
webclient := &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
LocalAddr: &localTCPAddr,
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
DualStack: true,
}).DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
},
}
Here is a fully working example that incorporates the answer from Tim. I also broke out all of the nested pieces to make it easier to read and learn from.
package main
import (
"fmt"
"io/ioutil"
"net"
"net/http"
"time"
)
func main() {
localAddr, err := net.ResolveIPAddr("ip", "10.128.64.219")
if err != nil {
panic(err)
}
localTCPAddr := net.TCPAddr{
IP: localAddr.IP,
}
d := net.Dialer{
LocalAddr: &localTCPAddr,
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
tr := &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: d.Dial,
TLSHandshakeTimeout: 10 * time.Second,
}
webclient := &http.Client{Transport: tr}
// Use NewRequest so we can change the UserAgent string in the header
req, err := http.NewRequest("GET", "http://www.google.com:80", nil)
if err != nil {
panic(err)
}
res, err := webclient.Do(req)
if err != nil {
panic(err)
}
fmt.Println("DEBUG", res)
defer res.Body.Close()
content, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
fmt.Printf("%s", string(content))
}
I wrote a golang program which run well in the past several months in ubuntu 12.04 LTS until I upgraded it to 14.04 LTS
My program is focused on sending HTTP requests which send about 2-10 HTTP requests per second. The HTTP request address vary.
When the problem occurs, first, some of the requests shows read tcp [ip]:[port]: i/o timeout, then after several minutes all requests show read tcp [ip]:[port]: i/o timeout, not any request can be sent.
I restart the program, everything become right again.
All of our servers(2 server) have such problem after upgraded from 12.04 to 14.04
I create new goroutine for every request
the problem does not occur in the same interval, sometimes it won't occur one or two day, sometimes It occur twice in an hour
Bellow is my code requesting HTTP Address:
t := &http.Transport{
Dial: timeoutDial(data.Timeout),
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
//req := s.ParseReq(data)
req := data.convert2Request()
if req == nil {
return
}
var resp *http.Response
if data.Redirect {
c := &http.Client{
Transport: t,
}
resp, err = c.Do(req)
} else {
resp, err = t.RoundTrip(req)
}
data.updateTry()
r := s.ParseResp(data, resp, err)
updateTry:
func (d *SendData) updateTry() {
d.Try++
d.LastSend = time.Now()
}
timeoutDial:
func timeoutDial(timeout int) func(netw, addr string) (net.Conn, error) {
if timeout <= 0 {
timeout = 10
}
return func(netw, addr string) (net.Conn, error) {
deadline := time.Now().Add(time.Duration(timeout) * time.Second)
c, err := net.DialTimeout(netw, addr, time.Second*time.Duration(timeout+5))
if err != nil {
return nil, err
}
c.SetDeadline(deadline)
return c, nil
}
}
and My dealing with response is:
func (s *Sender) ParseResp(data SendData, resp *http.Response, err error) (r Resp) {
r = Resp{URL: data.URL}
if err != nil {
r.Err = err.Error()
} else {
r.HttpCode = resp.StatusCode
r.Header = resp.Header
r.URL = resp.Request.URL.String()
defer resp.Body.Close()
// we just read part of response and log it.
reader := bufio.NewReader(resp.Body)
buf := make([]byte, bytes.MinRead) // 512 byte
for len(r.Body) < 1024 { // max 1k
var n int
if n, _ = reader.Read(buf); n == 0 {
break
}
r.Body += string(buf[:n])
}
}
return
}
I also found setting in /etc/sysctl.conf which can make the problem happen less frequently:
net.core.somaxconn = 65535
net.netfilter.nf_conntrack_max = 655350
net.netfilter.nf_conntrack_tcp_timeout_established = 1200
I need help for solving this problem.
It seems like this but I don't see any solution https://bugs.launchpad.net/juju-core/+bug/1307434
To more explicitly state what Not_a_Golfer and OneOfOne have said, when you're done with the response, you need to close the connection which has been left open (through the Body field which is an io.ReadCloser). So basically, one simple though would be to change the code pertaining to making an http request to:
var resp *http.Response
if data.Redirect {
c := &http.Client{
Transport: t,
}
resp, err = c.Do(req)
} else {
resp, err = t.RoundTrip(req)
}
if err == nil {
defer resp.Body.Close() // we need to close the connection
}
Without seeing the code to timeoutDial, my wild guess is that you don't close the connection when you're done with it.
Is there any other better way to ping websites and check if the website is available or not?
I just need to get the status code not get(download) all websites...
func Ping(domain string) int {
timeout := time.Duration(2 * time.Second)
dialTimeout := func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
url := "http://" + domain
req, _ := http.NewRequest("GET", url, nil)
resp, _ := client.Do(req)
return resp.StatusCode
}
This function is too slow and when I run with goroutines, it goes over the limits and gives me the errors...
Thanks!
Use a single transport. Because the transport maintains a pool of connections, you should not create and ignore transports willy nilly.
Close the response body as described at the beginning of the net/http doc.
Use HEAD if you are only interested in the status.
Check errors.
Code:
var client = http.Client{
Transport: &http.Transport{
Dial: net.Dialer{Timeout: 2 * time.Second}.Dial,
},
}
func Ping(domain string) (int, error) {
url := "http://" + domain
req, err := http.NewRequest("HEAD", url, nil)
if err != nil {
return 0, err
}
resp, err := client.Do(req)
if err != nil {
return 0, err
}
resp.Body.Close()
return resp.StatusCode, nil
}
Since this is the top result on Google for Pinging in Go, just know there have been several packages written for this purpose, but if you plan to use this answer, I had to make some changes for this to work.
import (
"time"
"net/http"
)
var client = http.Client{
Timeout: 2 * time.Second,
}
But otherwise keeping the same with the accepted answer.
But I'm a beginner in Go so there may be a better way to do this.