Check if a URL is reachable using Golang - http

I want to create a simple script that checks if a certain hostname:port is running. I only want to get a bool response if that URL is live, but I'm not sure if there's a straightforward way of doing it.

If you only want see if a URL is reachable you could use net.DialTimeout. Like this:
timeout := 1 * time.Second
conn, err := net.DialTimeout("tcp","mysyte:myport", timeout)
if err != nil {
log.Println("Site unreachable, error: ", err)
}

If you want to check if a Web server answers on a certain URL, you can invoke an HTTP GET request using net/http.
You will get a timeout if the server doesn't response at all. You might also check the response status.
resp, err := http.Get("http://google.com/")
if err != nil {
print(err.Error())
} else {
print(string(resp.StatusCode) + resp.Status)
}
You can change the default timeout by initializing a http.Client.
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
resp, err := client.Get("http://google.com")
Bonus:
Go generally does not rely on exceptions and the built in libraries generally do not panic, but return an error as a second value.
See Why does Go not have exceptions?.
You can assume that something very bad happened if your call to a native function panics.

You can make a HEAD request:
package main
import "net/http"
func head(s string) bool {
r, e := http.Head(s)
return e == nil && r.StatusCode == 200
}
func main() {
b := head("https://stackoverflow.com")
println(b)
}
https://golang.org/pkg/net/http#Head

If you don't mind the port, use http.Get(web):
package main
import (
"fmt"
"net/http"
"os"
)
func main() {
web := os.Args[1]
fmt.Println(webIsReachable(web))
}
func webIsReachable(web string) bool {
response, errors := http.Get(web)
if errors != nil {
_, netErrors := http.Get("https://www.google.com")
if netErrors != nil {
fmt.Fprintf(os.Stderr, "no internet\n")
os.Exit(1)
}
return false
}
if response.StatusCode == 200 {
return true
}
return false
}

Related

Pull info from private site that requires login (Golang)

I am trying to pull my classes from my online timetable, however, it seems as if I cannot get past the login stage. My code is:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"strings"
"github.com/PuerkitoBio/goquery"
)
type App struct {
Client *http.Client
}
type Timetable struct {
Name string
}
const (
baseURL string = "https://myclasswebsite.com"
)
func (app *App) login() {
//login := loginInfo()
client := app.Client
loginURL := baseURL + "/portal2/#!/login"
data := url.Values{
"inputEmail": {"my_actual_username"},
"password": {"my_actual_password"},
}
response, err := client.PostForm(loginURL, data)
if err != nil {
log.Fatalln(err)
}
defer response.Body.Close()
_, err = ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalln(err)
}
}
func (app *App) getTimetable() []Timetable {
timetableURL := baseURL + "/portal/dashboard"
client := app.Client
response, err := client.Get(timetableURL)
if err != nil {
log.Fatalln("Error fetching response. ", err)
}
defer response.Body.Close()
document, err := goquery.NewDocumentFromReader(response.Body)
fmt.Println(document.Html())
if err != nil {
log.Fatal("Error loading HTTP response body. ", err)
}
var classes []Timetable
document.Find(".timetable table").Each(func(i int, s *goquery.Selection) {
className := strings.TrimSpace(s.Text())
class := Timetable{
Name: className,
}
classes = append(classes, class)
})
return classes
}
I changed the base URL and login info just for privacy reasons, however, the rest of the code is as-is.
My main. go file is:
package main
import (
"fmt"
"net/http"
"net/http/cookiejar"
)
func main() {
jar, _ := cookiejar.New(nil)
app := App{
Client: &http.Client{Jar: jar},
}
app.login()
classes := app.getTimetable()
fmt.Println("class array is", classes)
for index, class := range classes {
fmt.Printf("%d: %s\n", index+1, class.Name)
}
}
The final print returns an empty slice, and when I print the response.Html() to the console, I receive the login-page Html rather than the dashboard-page HTML.
I'm in no way expecting anyone to fix this for me but a second pair of eyes and maybe a clue in which direction I should go would be helpful. Thank you so much!
Since I'm unfamiliar with your class website, some ideas for progressing:
You're not checking the status code from your login call. You may be getting a non-200 status code.
After you've confirmed the status code, check the cookie jar to ensure that a cookie has been saved. This is probably on the unlikelier side, but it's worth checking.
Lastly, attempt the same sequence with curl with -v (if you haven't already). That will give you more insight as to what is happening with your call plan.
I ended up ignoring some cookies accidentally while testing. Whoops...

Does the http request automatically retry?

I am trying to push my data to apache server using GoLang. Suppose my apache server is temporarily stopped. Then will my http request retry automatically.
I am using this statement
resp, err := http.DefaultClient.Do(req)
if err != nil {
return errors.Wrap(err, "http request error")
}
I am unable to proceed further coz what is think is my execution is stuck here. And I am repeatedly getting this error.
No, you will need to implement your own retry method, this is a basic example that could give you an idea:
https://play.golang.org/p/_o5AgePDEXq
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
var (
err error
response *http.Response
retries int = 3
)
for retries > 0 {
response, err = http.Get("https://non-existent")
// response, err = http.Get("https://google.com/robots.txt")
if err != nil {
log.Println(err)
retries -= 1
} else {
break
}
}
if response != nil {
defer response.Body.Close()
data, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("data = %s\n", data)
}
}
You can use more cleaner implementation of retrying by using retryablehttp
https://github.com/hashicorp/go-retryablehttp which handles most of the conditions
This is provides customisation for retry policy, backoffs etc.

Get gorilla/mux router current route name from middleware

Problem:
Unable to access mux.CurrentRoute(r).GetName() from middleware. (Although I had been able to access it from my middleware, I had to change the way my middleware works due to it's previous inability to access the request). So I've mucked something up and I'm not sure how to get back to a working state where I can access the route name.
Any help would be much appreciated!
Error:
runtime error: invalid memory address or nil pointer dereference
Code:
func main() {
var (
err error
r *mux.Router
devRouter *mux.Router
usersRouter *mux.Router
brandsRouter *mux.Router
)
defer db.Close()
defer store.Close()
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
usersRouter = r.PathPrefix("/api/v1/users").Subrouter()
brandsRouter = r.PathPrefix("/api/v1/brands").Subrouter()
// development endpoints
devRouter.HandleFunc("/db/seed", devDbSeed)
...
// users
usersRouter.HandleFunc("/create", usersCreateHandlerFunc).Methods("POST").Name("USERS_CREATE")
...
// brands
brandsRouter.HandleFunc("/create", brandsCreateHandlerFunc).Methods("POST").Name("BRANDS_CREATE")
...
// products
brandsRouter.HandleFunc("/{brand_id:[0-9]+}/products", brandsProductsListHandlerFunc).Methods("GET").Name("BRANDS_PRODUCTS_LIST")
...
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
http.Handle("/", use(r, mw...))
err = http.ListenAndServe(":9000", nil)
if err != nil {
logIt(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if true != authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
} else {
next.ServeHTTP(w, r)
}
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
/**
This is where it's failing!
*/
routeName := mux.CurrentRoute(r).GetName()
switch routeName {
case "USERS_CREATE":
// route-specific authorization
break
...
default:
break
}
return
}
Update (2015-01-04 # 4:49PM EST):
So after removing the middleware (or at least commenting out the section that's trying to read mux.CurrentRoute) I am able to retrieve the route name from the destination handlerfunc (ex: usersCreateHandlerFunc or brandsCreateHandlerFunc). This doesn't solve my problem (I'd still like to perform authentication/authorization in middleware as opposed to every handlerfunc), I have a hunch it's letting me know *mux.Router isn't available in my middleware until after the final .ServeHTTP call. (Or something along those lines...)
Update (2015-01-04 # 5:41PM EST):
Tried a different (albeit less-preferred) direction of using Negroni as the middleware component. Still getting nil-pointer error when I try to get mux.CurrentRoute.
Update (2015-01-04 # 6:17PM EST):
I am able to access the request (ex: r.URL) from the middleware func's, but still no luck on accessing the mux.Route (ex: mux.CurrentRoute(r)). After looking a bit more at the mux source, I think it's because the current mux context isn't getting set because the router hasn't executed the matcher yet (and therefore it doesn't know what route it's currently on until AFTER the middleware is complete). However, I'm still not sure how to either resolve this, or re-structure my code to handle this.
What about:
routeName := mux.CurrentRoute(r).GetName()
Where r is the *http.Request. Don't forget to import "github.com/gorilla/mux". Remember that in order to use this, you must give you route a name when you define it
From CurrentRoute godoc:
CurrentRoute returns the matched route for the current request, if any. This only works when called inside the handler of the matched route because the matched route is stored in the request context[...]
In your example, your chain of mwAuthenticate, mwAuthorize is attached to the route "/" without using gorilla mux. That means when the request passes your handlers, it has not passed gorilla mux router.
Try the following (your example stripped down):
package main
import (
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
)
var (
err error
r *mux.Router
devRouter *mux.Router
)
func devDbSeed(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "devDbSeed")
return
}
func main() {
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
// development endpoints
devRouter.Handle("/db/seed", use(http.HandlerFunc(devDbSeed), mw...)).Name("foo")
// Send all requests into the mux router
err = http.ListenAndServe(":9000", r)
if err != nil {
log.Fatal(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
})
}
func mwAuthenticate(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
next.ServeHTTP(w, r)
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
handlerName := "UNKNOWN"
if route := mux.CurrentRoute(r); route != nil {
routeName := route.GetName()
if routeName != "" {
handlerName = routeName
}
}
log.Println(handlerName)
switch handlerName {
case "USERS_CREATE":
// route-specific authorization
log.Println("USERS_CREATE")
break
default:
break
}
return
}
I had the same problem and I resolved in that way:
var match mux.RouteMatch
routeExists := s.Router.Match(r, &match)
if routeExists && match.Route.GetName(){
routeName := match.Route.GetName()
}
And when I defined the route I added .Name("route/:param") where route/:param is my route.

How Can I Make the Go HTTP Client NOT Follow Redirects Automatically?

I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/

Getting "127.0.0.1 can't assign requested address" - http.Client

What I'm doing is fairly straight-forward. I need to create a "proxy" server that is very minimal and fast. Currently I have a baseline server that is proxied to (nodejs) and a proxy-service (go). Please excuse the lack of actual "proxy'ing" - just testing for now.
Baseline Service
var http = require('http');
http.createServer(function (req, res) {
// console.log("received request");
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(8080, '127.0.0.1');
console.log('Server running at http://127.0.0.1:8080/');
Proxy Service
package main
import (
"flag"
"log"
"net/http"
"net/url"
)
var (
listen = flag.String("listen", "0.0.0.0:9000", "listen on address")
logp = flag.Bool("log", false, "enable logging")
)
func main() {
flag.Parse()
proxyHandler := http.HandlerFunc(proxyHandlerFunc)
log.Fatal(http.ListenAndServe(*listen, proxyHandler))
log.Println("Started router-server on 0.0.0.0:9000")
}
func proxyHandlerFunc(w http.ResponseWriter, r *http.Request) {
// Log if requested
if *logp {
log.Println(r.URL)
}
/*
* Tweak the request as appropriate:
* - RequestURI may not be sent to client
* - Set new URL
*/
r.RequestURI = ""
u, err := url.Parse("http://localhost:8080/")
if err != nil {
log.Fatal(err)
}
r.URL = u
// And proxy
// resp, err := client.Do(r)
c := make(chan *http.Response)
go doRequest(c)
resp := <-c
if resp != nil {
err := resp.Write(w)
if err != nil {
log.Println("Error writing response")
} else {
resp.Body.Close()
}
}
}
func doRequest(c chan *http.Response) {
// new client for every request.
client := &http.Client{}
resp, err := client.Get("http://127.0.0.1:8080/test")
if err != nil {
log.Println(err)
c <- nil
} else {
c <- resp
}
}
My issue, as mentioned within the title, is that I am getting errors stating 2013/10/28 21:22:30 Get http://127.0.0.1:8080/test: dial tcp 127.0.0.1:8080: can't assign requested address from the doRequest function, and I have no clue why. Googling this particular error yields seemingly irrelevant results.
There are 2 major problems with this code.
You are not handling the client stalling or using keep alives (handled below by getTimeoutServer)
You are not handling the server (what your http.Client is talking to) timing out (handled below by TimeoutConn).
This is probably why you are exhausting your local ports. I know from past experience node.js will keep-alive you very aggressively.
There are lots of little issues, creating objects every-time when you don't need to. Creating unneeded goroutines (each incoming request is in its own goroutine before you handle it).
Here is a quick stab (that I don't have time to test well). Hopefully it will put you on the right track: (You will want to upgrade this to not buffer the responses locally)
package main
import (
"bytes"
"errors"
"flag"
"fmt"
"log"
"net"
"net/http"
"net/url"
"runtime"
"strconv"
"time"
)
const DEFAULT_IDLE_TIMEOUT = 5 * time.Second
var (
listen string
logOn bool
localhost, _ = url.Parse("http://localhost:8080/")
client = &http.Client{
Transport: &http.Transport{
Proxy: NoProxyAllowed,
Dial: func(network, addr string) (net.Conn, error) {
return NewTimeoutConnDial(network, addr, DEFAULT_IDLE_TIMEOUT)
},
},
}
)
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
flag.StringVar(&listen, "listen", "0.0.0.0:9000", "listen on address")
flag.BoolVar(&logOn, "log", true, "enable logging")
flag.Parse()
server := getTimeoutServer(listen, http.HandlerFunc(proxyHandlerFunc))
log.Printf("Starting router-server on %s\n", listen)
log.Fatal(server.ListenAndServe())
}
func proxyHandlerFunc(w http.ResponseWriter, req *http.Request) {
if logOn {
log.Printf("%+v\n", req)
}
// Setup request URL
origURL := req.URL
req.URL = new(url.URL)
*req.URL = *localhost
req.URL.Path, req.URL.RawQuery, req.URL.Fragment = origURL.Path, origURL.RawQuery, origURL.Fragment
req.RequestURI, req.Host = "", req.URL.Host
// Perform request
resp, err := client.Do(req)
if err != nil {
w.WriteHeader(http.StatusBadGateway)
w.Write([]byte(fmt.Sprintf("%d - StatusBadGateway: %s", http.StatusBadGateway, err)))
return
}
defer resp.Body.Close()
var respBuffer *bytes.Buffer
if resp.ContentLength != -1 {
respBuffer = bytes.NewBuffer(make([]byte, 0, resp.ContentLength))
} else {
respBuffer = new(bytes.Buffer)
}
if _, err = respBuffer.ReadFrom(resp.Body); err != nil {
w.WriteHeader(http.StatusBadGateway)
w.Write([]byte(fmt.Sprintf("%d - StatusBadGateway: %s", http.StatusBadGateway, err)))
return
}
// Write result of request
headers := w.Header()
var key string
var val []string
for key, val = range resp.Header {
headers[key] = val
}
headers.Set("Content-Length", strconv.Itoa(respBuffer.Len()))
w.WriteHeader(resp.StatusCode)
w.Write(respBuffer.Bytes())
}
func getTimeoutServer(addr string, handler http.Handler) *http.Server {
//keeps people who are slow or are sending keep-alives from eating all our sockets
const (
HTTP_READ_TO = DEFAULT_IDLE_TIMEOUT
HTTP_WRITE_TO = DEFAULT_IDLE_TIMEOUT
)
return &http.Server{
Addr: addr,
Handler: handler,
ReadTimeout: HTTP_READ_TO,
WriteTimeout: HTTP_WRITE_TO,
}
}
func NoProxyAllowed(request *http.Request) (*url.URL, error) {
return nil, nil
}
//TimeoutConn-------------------------
//Put me in my own TimeoutConn.go ?
type TimeoutConn struct {
net.Conn
readTimeout, writeTimeout time.Duration
}
var invalidOperationError = errors.New("TimeoutConn does not support or allow .SetDeadline operations")
func NewTimeoutConn(conn net.Conn, ioTimeout time.Duration) (*TimeoutConn, error) {
return NewTimeoutConnReadWriteTO(conn, ioTimeout, ioTimeout)
}
func NewTimeoutConnReadWriteTO(conn net.Conn, readTimeout, writeTimeout time.Duration) (*TimeoutConn, error) {
this := &TimeoutConn{
Conn: conn,
readTimeout: readTimeout,
writeTimeout: writeTimeout,
}
now := time.Now()
err := this.Conn.SetReadDeadline(now.Add(this.readTimeout))
if err != nil {
return nil, err
}
err = this.Conn.SetWriteDeadline(now.Add(this.writeTimeout))
if err != nil {
return nil, err
}
return this, nil
}
func NewTimeoutConnDial(network, addr string, ioTimeout time.Duration) (net.Conn, error) {
conn, err := net.DialTimeout(network, addr, ioTimeout)
if err != nil {
return nil, err
}
if conn, err = NewTimeoutConn(conn, ioTimeout); err != nil {
return nil, err
}
return conn, nil
}
func (this *TimeoutConn) Read(data []byte) (int, error) {
this.Conn.SetReadDeadline(time.Now().Add(this.readTimeout))
return this.Conn.Read(data)
}
func (this *TimeoutConn) Write(data []byte) (int, error) {
this.Conn.SetWriteDeadline(time.Now().Add(this.writeTimeout))
return this.Conn.Write(data)
}
func (this *TimeoutConn) SetDeadline(time time.Time) error {
return invalidOperationError
}
func (this *TimeoutConn) SetReadDeadline(time time.Time) error {
return invalidOperationError
}
func (this *TimeoutConn) SetWriteDeadline(time time.Time) error {
return invalidOperationError
}
We ran into this and after a lot of time trying to debug, I came across this: https://code.google.com/p/go/source/detail?r=d4e1ec84876c
This shifts the burden onto clients to read their whole response
bodies if they want the advantage of reusing TCP connections.
So be sure you read the entire body before closing, there are a couple of ways to do it. This function can come in handy to close to let you see whether you have this issue by logging the extra bytes that haven't been read and cleaning the stream out for you so it can reuse the connection:
func closeResponse(response *http.Response) error {
// ensure we read the entire body
bs, err2 := ioutil.ReadAll(response.Body)
if err2 != nil {
log.Println("Error during ReadAll!!", err2)
}
if len(bs) > 0 {
log.Println("Had to read some bytes, not good!", bs, string(bs))
}
return response.Body.Close()
}
Or if you really don't care about the body, you can just discard it with this:
io.Copy(ioutil.Discard, response.Body)
I have encountered this problem too, and i add an option {DisableKeepAlives: true} to http.Transport fixed this issue, you can have a try.
I came here when running a massive amount of SQL queries per second on a system without limiting the number of idle connections over a long period of time. As pointed out in this issue comment on github explicitly setting db.SetMaxIdleConns(5) completely solved my problem.

Resources