I'd like to log 301s vs 302s but can't see a way to read the response status code in Client.Do, Get, doFollowingRedirects, CheckRedirect. Will I have to implement redirection myself to achieve this?
The http.Client type allows you to specify a custom transport, which should allow you to do what you're after. Something like the following should do:
type LogRedirects struct {
Transport http.RoundTripper
}
func (l LogRedirects) RoundTrip(req *http.Request) (resp *http.Response, err error) {
t := l.Transport
if t == nil {
t = http.DefaultTransport
}
resp, err = t.RoundTrip(req)
if err != nil {
return
}
switch resp.StatusCode {
case http.StatusMovedPermanently, http.StatusFound, http.StatusSeeOther, http.StatusTemporaryRedirect:
log.Println("Request for", req.URL, "redirected with status", resp.StatusCode)
}
return
}
(you could simplify this a little if you only support chaining to the default transport).
You can then create a client using this transport, and any redirects should be logged:
client := &http.Client{Transport: LogRedirects{}}
Here is a full example you can experiment with: http://play.golang.org/p/8uf8Cn31HC
Related
I have the following simple http.Client:
import (
"net/http"
"log"
)
...
func main() {
...
link = "http://example.com"
method = "GET"
req, _ := http.NewRequest(method, link, nil)
client := &http.Client{}
myZapLogger.Info("Sending a %s request to %s\n", method, link)
resp, err := client.Do(req)
if err != nil {
myZapLogger.Error(..., err) // I'm logging rather than fatal-ing or so
} else {
myZapLogger.Info("Received a %d on request X", resp.StatusCode)
}
...
}
...
I was looking for a way to do the above for each request through a hook (or so), so that it's triggered automatically each time. I can write a function the encloses all that, but in a case where I'm passing an http client to some other package, I wouldn't be able to control/log such requests that way (e.g. aws-go-sdk).
Is there a way to do this through contexts or attaching hooks to the client?
Thanks
eudore's comment answers the question; I'll just put it into code:
type MyRoundTripper struct {}
func (t MyRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
// Do work before the request is sent
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
// Do work after the response is received
return resp, err
}
To use it, you'll just pass it to your HTTP Client:
rt := MyRoundTripper{}
client := http.Client{Transport: rt}
Intuitively, I would think that when you create a MaxByteReader and pass in the http.ResponseWriter, it would write out the status code for you. But that isn't the case, what does the writer actually do?
example:
func maxBytesMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, 1)
next.ServeHTTP(w, r)
})
}
func mainHandler(w http.ResponseWriter, r *http.Request) {
var i interface{}
err := json.NewDecoder(r.Body).Decode(&i)
if err != nil {
fmt.Println(err.Error())
}
}
func TestMaxBytesMiddleware(t *testing.T) {
handlerToTest := maxBytesMiddleware(http.HandlerFunc(mainHandler))
req := httptest.NewRequest(http.MethodPost, "http://test.com", bytes.NewReader(json.RawMessage(`{"hello":"world"}`)))
recorder := httptest.NewRecorder()
handlerToTest.ServeHTTP(recorder, req)
if recorder.Result().StatusCode != http.StatusRequestEntityTooLarge {
t.Errorf("expected %d got %d", http.StatusRequestEntityTooLarge, recorder.Result().StatusCode)
}
}
but when this test runs I get this:
http: request body too large
--- FAIL: TestMaxBytesMiddleware (0.00s)
main_test.go:37: expected 413 got 200
if I want the desired functionality of what I thought this function did, I need to change my mainHandler to something like this:
func mainHandler(w http.ResponseWriter, r *http.Request) {
var i interface{}
err := json.NewDecoder(r.Body).Decode(&i)
if err != nil {
if err.Error() == "http: request body too large" {
w.WriteHeader(http.StatusRequestEntityTooLarge)
return
}
fmt.Println(err.Error())
}
}
So what is that writer even there for?
If the MaxBytesReader stops before reading the whole body, it sets some flags on the writer that make sure that the HTTP connection will be closed after the response is sent. Normally the server would be willing to read another request from the same connection (HTTP keepalive), but it can't do that if there are unread bits of the previous request still in the pipeline, so it has to close the connection, forcing the client to make a new connection if it wants to send more requests.
This is accomplished using the private requestTooLarge method of http.ResponseWriter.
Go version: go1.8.1 windows/amd64
Sample code for HTTP request is:
func (c *Client) RoundTripSoap12(action string, in, out Message) error {
fmt.Println("****************************************************************")
headerFunc := func(r *http.Request) {
r.Header.Add("Content-Type", fmt.Sprintf("text/xml; charset=utf-8"))
r.Header.Add("SOAPAction", fmt.Sprintf(action))
r.Cookies()
}
return doRoundTrip(c, headerFunc, in, out)
}
func doRoundTrip(c *Client, setHeaders func(*http.Request), in, out Message) error {
req := &Envelope{
EnvelopeAttr: c.Envelope,
NSAttr: c.Namespace,
Header: c.Header,
Body: Body{Message: in},
}
if req.EnvelopeAttr == "" {
req.EnvelopeAttr = "http://schemas.xmlsoap.org/soap/envelope/"
}
if req.NSAttr == "" {
req.NSAttr = c.URL
}
var b bytes.Buffer
err := xml.NewEncoder(&b).Encode(req)
if err != nil {
return err
}
cli := c.Config
if cli == nil {
cli = http.DefaultClient
}
r, err := http.NewRequest("POST", c.URL, &b)
if err != nil {
return err
}
setHeaders(r)
if c.Pre != nil {
c.Pre(r)
}
fmt.Println("*************", r)
resp, err := cli.Do(r)
if err != nil {
fmt.Println("error occured is as follows ", err)
return err
}
fmt.Println("response headers are: ", resp.Header.Get("sprequestguid"))
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
// read only the first Mb of the body in error case
limReader := io.LimitReader(resp.Body, 1024*1024)
body, _ := ioutil.ReadAll(limReader)
return fmt.Errorf("%q: %q", resp.Status, body)
}
return xml.NewDecoder(resp.Body).Decode(out)
I will call the RoundTripSoap12 function on the corresponding HTTP client.
When I send a request for the first time I will be getting some headers in the HTTP response, so these HTTP response headers should be sent as-is in my next HTTP request.
You may be interested in the httputil package and the reverse proxy example provided if you wish to proxy requests transparently:
https://golang.org/src/net/http/httputil/reverseproxy.go
You can copy the headers from one request to another one fairly easily - the Header is a separate object, if r and rc are http.Requests and you don't mind them sharing a header (you may need to clone instead if you want independent requests):
rc.Header = r.Header // note shallow copy
fmt.Println("Headers", r.Header, rc.Header)
https://play.golang.org/p/q2KUHa_qiP
Or you can look through keys and values and only copy certain headers, and/or do a clone instead to ensure you share no memory. See the http util package here for examples of this - see the functions cloneHeader and copyHeader inside reverseproxy.go linked above.
I want to create a simple script that checks if a certain hostname:port is running. I only want to get a bool response if that URL is live, but I'm not sure if there's a straightforward way of doing it.
If you only want see if a URL is reachable you could use net.DialTimeout. Like this:
timeout := 1 * time.Second
conn, err := net.DialTimeout("tcp","mysyte:myport", timeout)
if err != nil {
log.Println("Site unreachable, error: ", err)
}
If you want to check if a Web server answers on a certain URL, you can invoke an HTTP GET request using net/http.
You will get a timeout if the server doesn't response at all. You might also check the response status.
resp, err := http.Get("http://google.com/")
if err != nil {
print(err.Error())
} else {
print(string(resp.StatusCode) + resp.Status)
}
You can change the default timeout by initializing a http.Client.
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
resp, err := client.Get("http://google.com")
Bonus:
Go generally does not rely on exceptions and the built in libraries generally do not panic, but return an error as a second value.
See Why does Go not have exceptions?.
You can assume that something very bad happened if your call to a native function panics.
You can make a HEAD request:
package main
import "net/http"
func head(s string) bool {
r, e := http.Head(s)
return e == nil && r.StatusCode == 200
}
func main() {
b := head("https://stackoverflow.com")
println(b)
}
https://golang.org/pkg/net/http#Head
If you don't mind the port, use http.Get(web):
package main
import (
"fmt"
"net/http"
"os"
)
func main() {
web := os.Args[1]
fmt.Println(webIsReachable(web))
}
func webIsReachable(web string) bool {
response, errors := http.Get(web)
if errors != nil {
_, netErrors := http.Get("https://www.google.com")
if netErrors != nil {
fmt.Fprintf(os.Stderr, "no internet\n")
os.Exit(1)
}
return false
}
if response.StatusCode == 200 {
return true
}
return false
}
I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/