I am trying to push my data to apache server using GoLang. Suppose my apache server is temporarily stopped. Then will my http request retry automatically.
I am using this statement
resp, err := http.DefaultClient.Do(req)
if err != nil {
return errors.Wrap(err, "http request error")
}
I am unable to proceed further coz what is think is my execution is stuck here. And I am repeatedly getting this error.
No, you will need to implement your own retry method, this is a basic example that could give you an idea:
https://play.golang.org/p/_o5AgePDEXq
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
var (
err error
response *http.Response
retries int = 3
)
for retries > 0 {
response, err = http.Get("https://non-existent")
// response, err = http.Get("https://google.com/robots.txt")
if err != nil {
log.Println(err)
retries -= 1
} else {
break
}
}
if response != nil {
defer response.Body.Close()
data, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("data = %s\n", data)
}
}
You can use more cleaner implementation of retrying by using retryablehttp
https://github.com/hashicorp/go-retryablehttp which handles most of the conditions
This is provides customisation for retry policy, backoffs etc.
Related
I have the following simple http.Client:
import (
"net/http"
"log"
)
...
func main() {
...
link = "http://example.com"
method = "GET"
req, _ := http.NewRequest(method, link, nil)
client := &http.Client{}
myZapLogger.Info("Sending a %s request to %s\n", method, link)
resp, err := client.Do(req)
if err != nil {
myZapLogger.Error(..., err) // I'm logging rather than fatal-ing or so
} else {
myZapLogger.Info("Received a %d on request X", resp.StatusCode)
}
...
}
...
I was looking for a way to do the above for each request through a hook (or so), so that it's triggered automatically each time. I can write a function the encloses all that, but in a case where I'm passing an http client to some other package, I wouldn't be able to control/log such requests that way (e.g. aws-go-sdk).
Is there a way to do this through contexts or attaching hooks to the client?
Thanks
eudore's comment answers the question; I'll just put it into code:
type MyRoundTripper struct {}
func (t MyRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
// Do work before the request is sent
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
// Do work after the response is received
return resp, err
}
To use it, you'll just pass it to your HTTP Client:
rt := MyRoundTripper{}
client := http.Client{Transport: rt}
I want to create a simple script that checks if a certain hostname:port is running. I only want to get a bool response if that URL is live, but I'm not sure if there's a straightforward way of doing it.
If you only want see if a URL is reachable you could use net.DialTimeout. Like this:
timeout := 1 * time.Second
conn, err := net.DialTimeout("tcp","mysyte:myport", timeout)
if err != nil {
log.Println("Site unreachable, error: ", err)
}
If you want to check if a Web server answers on a certain URL, you can invoke an HTTP GET request using net/http.
You will get a timeout if the server doesn't response at all. You might also check the response status.
resp, err := http.Get("http://google.com/")
if err != nil {
print(err.Error())
} else {
print(string(resp.StatusCode) + resp.Status)
}
You can change the default timeout by initializing a http.Client.
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
resp, err := client.Get("http://google.com")
Bonus:
Go generally does not rely on exceptions and the built in libraries generally do not panic, but return an error as a second value.
See Why does Go not have exceptions?.
You can assume that something very bad happened if your call to a native function panics.
You can make a HEAD request:
package main
import "net/http"
func head(s string) bool {
r, e := http.Head(s)
return e == nil && r.StatusCode == 200
}
func main() {
b := head("https://stackoverflow.com")
println(b)
}
https://golang.org/pkg/net/http#Head
If you don't mind the port, use http.Get(web):
package main
import (
"fmt"
"net/http"
"os"
)
func main() {
web := os.Args[1]
fmt.Println(webIsReachable(web))
}
func webIsReachable(web string) bool {
response, errors := http.Get(web)
if errors != nil {
_, netErrors := http.Get("https://www.google.com")
if netErrors != nil {
fmt.Fprintf(os.Stderr, "no internet\n")
os.Exit(1)
}
return false
}
if response.StatusCode == 200 {
return true
}
return false
}
I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/
i use
resp, err := http.Get("http://example.com/")
get a http.Response, and i want to exactly write to a http handler, but only http.ResponseWriter, so i hijack it.
...
webConn, webBuf, err := hj.Hijack()
if err != nil {
// handle error
}
defer webConn.Close()
// Write resp
resp.Write(webBuf)
...
Write raw request
But When i hijack, http connection can't reuse (keep-alive), so it slow.
How to solve?
Thanks! Sorry for my pool English.
update 12/9
keep-alive, It keep two tcp connection, and can reuse.
but when i hijack, and conn.Close(), It can't reuse old connection, so it create a new tcp connection when i each refresh.
Do not use hijack, Because once hijack, the HTTP server library will not do anything else with the connection, So can't reuse.
I change way, copy Header and Body, look like reverse proxy (http://golang.org/src/pkg/net/http/httputil/reverseproxy.go), Is works.
Example:
func copyHeader(dst, src http.Header) {
for k, w := range src {
for _, v := range w {
dst.Add(k, v)
}
}
}
func copyResponse(r *http.Response, w http.ResponseWriter) {
copyHeader(w.Header(), r.Header)
w.WriteHeader(r.StatusCode)
io.Copy(w, r.Body)
}
func handler(w http.ResponseWriter, r *http.Response) {
resp, err := http.Get("http://www.example.com")
if err != nil {
// handle error
}
copyResponse(resp, w)
}
It seem that once the connection is closed the keep-alive connection closes as well.
One possible solution would be to prevent the connection from closing until desired, but I'm not sure if that good advise.
Maybe the correct solution involves creating a instance of net.TCPConn, copying the connection over it, then calling .SetKeepAlive(true).
Before running the below example, launch another terminal with netstat -antc | grep 9090.
Routes in example:
localhost:9090/ok is a basic (non-hijacked) connection
localhost:9090 is a hijacked connection, lasting for 10 seconds.
Example
package main
import (
"fmt"
"net/http"
"sync"
"time"
)
func checkError(e error) {
if e != nil {
panic(e)
}
}
var ka_seconds = 10
var conn_id = 0
func main() {
http.HandleFunc("/ok", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "ok")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
conn_id++
fmt.Printf("Connection %v: Keep-alive is enabled %v seconds\n", conn_id, ka_seconds)
hj, ok := w.(http.Hijacker)
if !ok {
http.Error(w, "webserver doesn't support hijacking", http.StatusInternalServerError)
return
}
conn, bufrw, err := hj.Hijack()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Don't forget to close the connection:
time.AfterFunc(time.Second* time.Duration(ka_seconds), func() {
conn.Close()
fmt.Printf("Connection %v: Keep-alive is disabled.\n", conn_id)
})
resp, err := http.Get("http://www.example.com")
checkError(err)
resp.Write(bufrw)
bufrw.Flush()
})
fmt.Println("Listing to localhost:9090")
http.ListenAndServe(":9090", nil)
}
Related issue: http://code.google.com/p/go/issues/detail?id=5645
I'm trying to detect sending failures by inspecting the error returned by golang TCPConn.Write, but it's nil. I also tried using TCPConn.SetWriteDeadline without success.
That's how things happen:
the server starts
a client connects
the server sends a message and the client receives it
the client shuts down
the server sends one more message: no error
the server sends the third message: only now the error appears
Question: why only the second message to a non-existing client results in an error? How should the case be handled properly?
The code follows:
package main
import (
"net"
"os"
"bufio"
"fmt"
"time"
)
func AcceptConnections(listener net.Listener, console <- chan string) {
msg := ""
for {
conn, err := listener.Accept()
if err != nil {
panic(err)
}
fmt.Printf("client connected\n")
for {
if msg == "" {
msg = <- console
fmt.Printf("read from console: %s", msg)
}
err = conn.SetWriteDeadline(time.Now().Add(time.Second))
if err != nil {
fmt.Printf("SetWriteDeadline failed: %v\n", err)
}
_, err = conn.Write([]byte(msg))
if err != nil {
// expecting an error after sending a message
// to a non-existing client endpoint
fmt.Printf("failed sending a message to network: %v\n", err)
break
} else {
fmt.Printf("msg sent: %s", msg)
msg = ""
}
}
}
}
func ReadConsole(network chan <- string) {
console := bufio.NewReader(os.Stdin)
for {
line, err := console.ReadString('\n')
if err != nil {
panic(err)
} else {
network <- line
}
}
}
func main() {
listener, err := net.Listen("tcp", "localhost:6666")
if err != nil {
panic(err)
}
println("listening on " + listener.Addr().String())
consoleToNetwork := make(chan string)
go AcceptConnections(listener, consoleToNetwork)
ReadConsole(consoleToNetwork)
}
The server console looks like this:
listening on 127.0.0.1:6666
client connected
hi there!
read from console: hi there!
msg sent: hi there!
this one should fail
read from console: this one should fail
msg sent: this one should fail
this one actually fails
read from console: this one actually fails
failed sending a message to network: write tcp 127.0.0.1:51194: broken pipe
The client looks like this:
package main
import (
"net"
"os"
"io"
//"bufio"
//"fmt"
)
func cp(dst io.Writer, src io.Reader, errc chan<- error) {
// -reads from src and writes to dst
// -blocks until EOF
// -EOF is not an error
_, err := io.Copy(dst, src)
// push err to the channel when io.Copy returns
errc <- err
}
func StartCommunication(conn net.Conn) {
//create a channel for errors
errc := make(chan error)
//read connection and print to console
go cp(os.Stdout, conn, errc)
//read user input and write to connection
go cp(conn, os.Stdin, errc)
//wait until nil or an error arrives
err := <- errc
if err != nil {
println("cp error: ", err.Error())
}
}
func main() {
servAddr := "localhost:6666"
tcpAddr, err := net.ResolveTCPAddr("tcp", servAddr)
if err != nil {
println("ResolveTCPAddr failed:", err.Error())
os.Exit(1)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
println("net.DialTCP failed:", err.Error())
os.Exit(1)
}
defer conn.Close()
StartCommunication(conn)
}
EDIT: Following JimB's suggestion I came up with a working example. Messages don't get lost any more and are re-sent in a new connection. I'm not quite sure though how safe is it to use a shared variable (connWrap.IsFaulted) between different go routines.
package main
import (
"net"
"os"
"bufio"
"fmt"
)
type Connection struct {
IsFaulted bool
Conn net.Conn
}
func StartWritingToNetwork(connWrap * Connection, errChannel chan <- error, msgStack chan string) {
for {
msg := <- msgStack
if connWrap.IsFaulted {
//put it back for another connection
msgStack <- msg
return
}
_, err := connWrap.Conn.Write([]byte(msg))
if err != nil {
fmt.Printf("failed sending a message to network: %v\n", err)
connWrap.IsFaulted = true
msgStack <- msg
errChannel <- err
return
} else {
fmt.Printf("msg sent: %s", msg)
}
}
}
func StartReadingFromNetwork(connWrap * Connection, errChannel chan <- error){
network := bufio.NewReader(connWrap.Conn)
for (!connWrap.IsFaulted) {
line, err := network.ReadString('\n')
if err != nil {
fmt.Printf("failed reading from network: %v\n", err)
connWrap.IsFaulted = true
errChannel <- err
} else {
fmt.Printf("%s", line)
}
}
}
func AcceptConnections(listener net.Listener, console chan string) {
errChannel := make(chan error)
for {
conn, err := listener.Accept()
if err != nil {
panic(err)
}
fmt.Printf("client connected\n")
connWrap := Connection{false, conn}
go StartReadingFromNetwork(&connWrap, errChannel)
go StartWritingToNetwork(&connWrap, errChannel, console)
//block until an error occurs
<- errChannel
}
}
func ReadConsole(network chan <- string) {
console := bufio.NewReader(os.Stdin)
for {
line, err := console.ReadString('\n')
if err != nil {
panic(err)
} else {
network <- line
}
}
}
func main() {
listener, err := net.Listen("tcp", "localhost:6666")
if err != nil {
panic(err)
}
println("listening on " + listener.Addr().String())
consoleToNetwork := make(chan string)
go AcceptConnections(listener, consoleToNetwork)
ReadConsole(consoleToNetwork)
}
This isn't Go specific, and is a artifact of the underlying TCP socket showing through.
A decent diagram of the TCP termination steps is at the bottom of this page:
http://www.tcpipguide.com/free/t_TCPConnectionTermination-2.htm
The simple version is that when the client closes its socket, it sends a FIN, and receives an ACK from the server. It then waits for the server to do the same. Instead of sending a FIN though, you're sending more data, which is discarded, and the client socket now assumes that any more data coming from you is invalid, so the next time you send you get an RST, which is what bubbles up into the error you see.
Going back to your program, you need to handle this somehow. Generally you can think of whomever is in charge of initiating a send, is also in charge of initiating termination, hence your server should assume that it can continue to send until it closes the connection, or encounters an error. If you need to more reliably detect the client closing, you need to have some sort of client response in the protocol. That way recv can be called on the socket and return 0, which alerts you to the closed connection.
In go, this will return an EOF error from the connection's Read method (or from within the Copy in your case). SetWriteDeadline doesn't work because a small write will go though and get dropped silently, or the client will eventually respond with an RST, giving you an error.