I have a client machine with multiple NICs, how do I bind an http.Client in Go to a certain NIC or to a certain SRC IP Address?
Say you have some very basic http client code that looks like:
package main
import (
"net/http"
)
func main() {
webclient := &http.Client{}
req, _ := http.NewRequest("GET", "http://www.google.com", nil)
httpResponse, _ := webclient.Do(req)
defer httpResponse.Body.Close()
}
Is there a way to bind to a certain NIC or IP?
Similar to this question, you need to set the http.Client.Transport field. Setting it to an instance of net.Transport allows you to specify which net.Dialer you want to use. net.Dialer then allows you to specify the local address to make connections from.
Example:
localAddr, err := net.ResolveIPAddr("ip", "<my local address>")
if err != nil {
panic(err)
}
// You also need to do this to make it work and not give you a
// "mismatched local address type ip"
// This will make the ResolveIPAddr a TCPAddr without needing to
// say what SRC port number to use.
localTCPAddr := net.TCPAddr{
IP: localAddr.IP,
}
webclient := &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
LocalAddr: &localTCPAddr,
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
DualStack: true,
}).DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
},
}
Here is a fully working example that incorporates the answer from Tim. I also broke out all of the nested pieces to make it easier to read and learn from.
package main
import (
"fmt"
"io/ioutil"
"net"
"net/http"
"time"
)
func main() {
localAddr, err := net.ResolveIPAddr("ip", "10.128.64.219")
if err != nil {
panic(err)
}
localTCPAddr := net.TCPAddr{
IP: localAddr.IP,
}
d := net.Dialer{
LocalAddr: &localTCPAddr,
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
tr := &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: d.Dial,
TLSHandshakeTimeout: 10 * time.Second,
}
webclient := &http.Client{Transport: tr}
// Use NewRequest so we can change the UserAgent string in the header
req, err := http.NewRequest("GET", "http://www.google.com:80", nil)
if err != nil {
panic(err)
}
res, err := webclient.Do(req)
if err != nil {
panic(err)
}
fmt.Println("DEBUG", res)
defer res.Body.Close()
content, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
fmt.Printf("%s", string(content))
}
Related
The http.Request struct includes the remote IP and port of the request's sender:
// RemoteAddr allows HTTP servers and other software to record
// the network address that sent the request, usually for
// logging. This field is not filled in by ReadRequest and
// has no defined format. The HTTP server in this package
// sets RemoteAddr to an "IP:port" address before invoking a
// handler.
// This field is ignored by the HTTP client.
**RemoteAddr string**
The http.Response object has no such field.
I would like to know the IP address that responded to the request I sent, even when I sent it to a DNS address.
I thought that net.LookupHost() might be helpful, but 1) it can return multiple IPs for a single host name, and 2) it ignores the hosts file unless cgo is available, which it is not in my case.
Is it possible to retrieve the remote IP address for an http.Response?
Use the net/http/httptrace package and use the GotConnInfo hook to capture the net.Conn and its corresponding Conn.RemoteAddr().
This will give you the address the Transport actually dialled, as opposed to what was resolved in DNSDoneInfo:
package main
import (
"log"
"net/http"
"net/http/httptrace"
)
func main() {
req, err := http.NewRequest("GET", "https://example.com/", nil)
if err != nil {
log.Fatal(err)
}
trace := &httptrace.ClientTrace{
GotConn: func(connInfo httptrace.GotConnInfo) {
log.Printf("resolved to: %s", connInfo.Conn.RemoteAddr())
},
}
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
client := &http.Client{}
_, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
}
Outputs:
~ go run ip.go
2017/02/18 19:38:11 resolved to: 104.16.xx.xxx:443
Another solution I came up with was the hook the DialContext function in the http client transport. This is a specific solution that lets you modify the http.Client instead of the request which may be useful.
We first create a function that returns a hooked dial context
func remoteAddressDialHook(remoteAddressPtr *net.Addr) func(ctx context.Context, network string, address string) (net.Conn, error) {
hookedDialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
originalDialer := &net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
conn, err := originalDialer.DialContext(ctx, network, address)
if err != nil {
return nil, err
}
// conn was successfully created
*remoteAddressPtr = conn.RemoteAddr()
return conn, err
}
return hookedDialContext
}
We can then use this function to create a DialContext that writes to an outparameter
var remoteAddr net.Addr
customTransport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: remoteAddressDialHook(&remoteAddr),
ForceAttemptHTTP2: true,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
customHttpClient := http.Client{
Transport: customTransport,
}
// do what you normally would with a http client, it will then set the remoteAddr to be the remote address
fmt.Println(remoteAddr.String())
I am a newbie to go language. I plan to use go language for developing an http client/Server.
While browsing through the list of features supported in http client package I do not find a way to set socket option in the package
(May be I just do not know how to use it). I am in a need to set DSCP option (IP_TOS) in the fd before calling http client connection.
(Although I find syscall option to set socket options, I do not find a way to get fd from http package).
In http server side, able to set socket option (IP_TOS).
code excerpt:
tcpListener,err := net.ListenTCP("tcp4", addr)
if err != nil {
//fmt.Println("error in listen", err.error())
log.Fatal("net.ListenTCP()", err)
}
//get lisenet socket fd
f, _ := tcpListener.File()
err = syscall.SetsockoptInt(int(f.Fd()), syscall.SOL_SOCKET, syscall.IP_TOS, 128)
In http client side, not able to get socket fd and set socket option (IP_TOS):
(I want to set IP_TOS before calling NewRequest)
client := &http.Client{
Transport : tr,
//Timeout: time.Duration(10) * time.Second,
}
request, err := http.NewRequest("POST", url, body)
if err != nil {
panic(err)
}
response, err := client.Do(request)
Thanks !!
You can create your own DialContext for your own http.RoundTripper of your own *http.Client:
dial := func(ctx context.Context, network, addr string) (net.Conn, error) {
conn, err := (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext(ctx, network, addr)
if err != nil {
return nil, err
}
tcpConn, ok := conn.(*net.TCPConn)
if !ok {
err = errors.New("conn is not tcp")
return nil, err
}
f, err := tcpConn.File()
if err != nil {
return nil, err
}
err = syscall.SetsockoptInt(int(f.Fd()), syscall.IPPROTO_IP, syscall.IP_TOS, 128)
if err != nil {
return nil, err
}
return conn, nil
}
tr := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: dial,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
c := &http.Client{
Transport: tr,
}
resp, err := c.Get("https://google.com/")
if err != nil {
log.Fatalf("GET error: %v", err)
}
log.Printf("got %q", resp.Status)
EDIT: If you need to set the option before the connection actually happens, you can try this in your DialContext, but this is fairly non-portable, unsafe, doesn't account for context, and will probably break sooner or later:
tcpAddr, err := net.ResolveTCPAddr(network, addr)
if err != nil {
return nil, err
}
sa := &syscall.SockaddrInet4{
Port: tcpAddr.Port,
Addr: [4]byte{tcpAddr.IP[0], tcpAddr.IP[1], tcpAddr.IP[2], tcpAddr.IP[3]},
}
fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)
if err != nil {
return nil, err
}
err = syscall.SetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_TOS, 128)
if err != nil {
return nil, err
}
err = syscall.Connect(fd, sa)
if err != nil {
return nil, err
}
file := os.NewFile(uintptr(fd), "")
conn, err := net.FileConn(file)
if err != nil {
return nil, err
}
return conn, nil
EDIT 2: In Go 1.11 you'll be able to do this:
dialer := &net.Dialer{
Control: func(network, address string, c syscall.RawConn) error {
return c.Control(func(fd uintptr) {
err := syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_TOS, 128)
if err != nil {
log.Printf("control: %s", err)
return
}
})
},
}
// ...
tr := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: dialer.DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
I have a Go program that generates a lot of HTTP requests from multiple goroutines. after running for a while, the program spits out an error: connect: cannot assign requested address.
When checking with netstat, I get a high number (28229) of connections in TIME_WAIT.
The high number of TIME_WAIT sockets happens when I the number of goroutines is 3 and is severe enough to cause a crash when it is 5.
I run Ubuntu 14.4 under docker and go version 1.7
This is the Go program.
package main
import (
"io/ioutil"
"log"
"net/http"
"sync"
)
var wg sync.WaitGroup
var url="http://172.17.0.9:3000/";
const num_coroutines=5;
const num_request_per_coroutine=100000
func get_page(){
response, err := http.Get(url)
if err != nil {
log.Fatal(err)
} else {
defer response.Body.Close()
_, err =ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
}
}
func get_pages(){
defer wg.Done()
for i := 0; i < num_request_per_coroutine; i++{
get_page();
}
}
func main() {
for i:=0;i<num_coroutines;i++{
wg.Add(1)
go get_pages()
}
wg.Wait()
}
This is the server program:
package main
import (
"fmt"
"net/http"
"log"
)
var count int;
func sayhelloName(w http.ResponseWriter, r *http.Request) {
count++;
fmt.Fprintf(w,"Hello World, count is %d",count) // send data to client side
}
func main() {
http.HandleFunc("/", sayhelloName) // set router
err := http.ListenAndServe(":3000", nil) // set listen port
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
The default http.Transport is opening and closing connections too quickly. Since all connections are to the same host:port combination, you need to increase MaxIdleConnsPerHost to match your value for num_coroutines. Otherwise, the transport will frequently close the extra connections, only to have them reopened immediately.
You can set this globally on the default transport:
http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = numCoroutines
Or when creating your own transport
t := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
MaxIdleConnsPerHost: numCoroutines,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
Similar question: Go http.Get, concurrency, and "Connection reset by peer"
I'm scraping HTML pages and have set up a HTTP client like so:
client := *http.Client{
Transport: &http.Transport{
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
ResponseHeaderTimeout: 10 * time.Second,
},
}
Now when I make GET requests of multiple URLs I don't want to get stuck with URLs that deliver massive amount of data.
response, err := client.Get(page.Url)
checkErr(err)
body, err := ioutil.ReadAll(response.Body)
checkErr(err)
page.Body = string(body)
Is there a way to limit the amount of data (bytes) the GET request accepts from a resource and stops?
Use an io.LimitedReader
A LimitedReader reads from R but limits the amount of data returned to just N bytes.
limitedReader := &io.LimitedReader{R: response.Body, N: limit}
body, err := ioutil.ReadAll(limitedReader)
or
body, err := ioutil.ReadAll(io.LimitReader(response.Body, limit))
You can use io.CopyN:
package main
import (
"io"
"net/http"
"os"
)
func main() {
r, e := http.Get("http://speedtest.lax.hivelocity.net")
if e != nil {
panic(e)
}
defer r.Body.Close()
io.CopyN(os.Stdout, r.Body, 100)
}
Or Range header:
package main
import (
"net/http"
"os"
)
func main() {
req, e := http.NewRequest("GET", "http://speedtest.lax.hivelocity.net", nil)
if e != nil {
panic(e)
}
req.Header.Set("Range", "bytes=0-99")
res, e := new(http.Client).Do(req)
if e != nil {
panic(e)
}
defer res.Body.Close()
os.Stdout.ReadFrom(res.Body)
}
https://golang.org/pkg/io#CopyN
https://golang.org/pkg/net/http#Request.Header
Is there any other better way to ping websites and check if the website is available or not?
I just need to get the status code not get(download) all websites...
func Ping(domain string) int {
timeout := time.Duration(2 * time.Second)
dialTimeout := func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
url := "http://" + domain
req, _ := http.NewRequest("GET", url, nil)
resp, _ := client.Do(req)
return resp.StatusCode
}
This function is too slow and when I run with goroutines, it goes over the limits and gives me the errors...
Thanks!
Use a single transport. Because the transport maintains a pool of connections, you should not create and ignore transports willy nilly.
Close the response body as described at the beginning of the net/http doc.
Use HEAD if you are only interested in the status.
Check errors.
Code:
var client = http.Client{
Transport: &http.Transport{
Dial: net.Dialer{Timeout: 2 * time.Second}.Dial,
},
}
func Ping(domain string) (int, error) {
url := "http://" + domain
req, err := http.NewRequest("HEAD", url, nil)
if err != nil {
return 0, err
}
resp, err := client.Do(req)
if err != nil {
return 0, err
}
resp.Body.Close()
return resp.StatusCode, nil
}
Since this is the top result on Google for Pinging in Go, just know there have been several packages written for this purpose, but if you plan to use this answer, I had to make some changes for this to work.
import (
"time"
"net/http"
)
var client = http.Client{
Timeout: 2 * time.Second,
}
But otherwise keeping the same with the accepted answer.
But I'm a beginner in Go so there may be a better way to do this.