Why is Go HTTPS Client not reusing connections? - http

Question

I have an http client which creates multiple connections to the host. I want to set a maximum number of connections it can set to a particular host.
There are no such options in go's request.Transport.
My code looks like
package main
import (
"fmt"
"net/http"
"net/url"
)
const (
endpoint_url_fmt = "https://blah.com/api1?%s"
)
func main() {
transport := http.Transport{ DisableKeepAlives : false }
outParams := url.Values{}
outParams.Set("method", "write")
outParams.Set("message", "BLAH")
for {
// Encode as part of URI.
outboundRequest, err := http.NewRequest(
"GET",
fmt.Sprintf(endpoint_url_fmt, outParams.Encode()),
nil
)
outboundRequest.Close = false
_ , err = transport.RoundTrip(outboundRequest)
if err != nil {
fmt.Println(err)
}
}
}
I would expect this to create 1 connection. As I am calling it in a for-loop. But this keeps creating an infinite number of connections.
Where as similar python code using the requests library creates only one connection.
#!/usr/bin/env python
import requests
endpoint_url_fmt = "https://something.com/restserver.php"
params = {}
params['method'] = 'write'
params['category'] = category_errors_scuba
params['message'] = "blah"
while True:
r = requests.get(endpoint_url_fmt, params = params)
For some reason the go code is not reusing http connections.
EDIT :
The go code needs the body to be closed to reuse the connection.
resp , err = transport.RoundTrip(outboundRequest)
resp.Close() // This allows the connection to be reused

Answer 1

Based on further clarification from the OP. The default client does reuse connections.
Be sure to close the response.
Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.
Additionally, I've found that I also needed to read until the response was complete before calling Close().
e.g.
res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()
To ensure http.Client connection reuse be sure to do two things:
Read until Response is complete (i.e. ioutil.ReadAll(resp.Body))
Call Body.Close()
Old answer, useful for rate limiting, but not what the OP was after:
I don't think setting max connections is possible via the golang 1.1 http APIs. This means you can shoot yourself in the foot with tons of TCP connections (until you run out of file descriptors or whatever) if you aren't careful.
That said, you could limit the rate at which you call the go routine for a particular host (and therefore outbound requests and connections) via time.Tick.
For example:
import "time"
requests_per_second := 5
throttle := time.Tick(1000000000 / requests_per_second)
for i := 0; i < 16; i += 1 {
<-throttle
go serveQueue()
}

Answer 2

There are interesting improvements in http.Transport:
// DisableKeepAlives, if true, disables HTTP keep-alives and
// will only use the connection to the server for a single
// HTTP request.
//
// This is unrelated to the similarly named TCP keep-alives.
DisableKeepAlives bool
// ...
// MaxIdleConns controls the maximum number of idle (keep-alive)
// connections across all hosts. Zero means no limit.
MaxIdleConns int // Go 1.7
// MaxIdleConnsPerHost, if non-zero, controls the maximum idle
// (keep-alive) connections to keep per-host. If zero,
// DefaultMaxIdleConnsPerHost is used.
MaxIdleConnsPerHost int
// MaxConnsPerHost optionally limits the total number of
// connections per host, including connections in the dialing,
// active, and idle states. On limit violation, dials will block.
//
// Zero means no limit.
MaxConnsPerHost int // Go 1.11