I have a handler which contains two goroutines. The handler is executed every 10 seconds by the frontend. Once called, the goroutines send a GET http request to an external API. For some reason, sometimes (not often) I randomly get the following error in any of the goroutines (seems that the external API refuses the request).
StatusCode:408
panic:read tcp 192.168.1.106:62598->80.243.175.58:443: wsarecv: An existing connection was forcibly closed by the remote host. goroutine
8280 [running]:
Monitoring/monitoring-v2-goAPI/services.GetNodeCpuLimits.func2(0x815ba0,
0xc04231a1c0, 0x0, 0x0, 0x0, 0x0, 0xc042158240, 0x14, 0x0, 0x0, ...)
When this happens the panic(err) is triggered and my application terminates. This error is happening since I rebuilt the handler in an asynchronous way to improve speed (before the requests were just synchronous).
I would like to know if im doing anything wrong. And if not, how can I handle this? I cant afford the application to terminate.
Code
func TestFunc(w http.ResponseWriter, r *http.Request) {
c := make(chan NgxJSON)
wg := &sync.WaitGroup{}
var queryParams QueryParams
_ = json.NewDecoder(r.Body).Decode(&queryParams)
fmt.Println("- queryParams ->", queryParams)
wg.Add(1)
go func(queryParams QueryParams, c chan NgxJSON, wg *sync.WaitGroup) {
defer wg.Done()
req, err := http.NewRequest("GET", APIURL1+queryParams.Param1, nil)
if err != nil {
fmt.Println(err)
return
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err) //<-- Error triggers here!!
}
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
var response Response
json.Unmarshal([]byte(string(body)), &response)
//DO THINGS
c <- ngxJSON
}(queryParams, c, wg)
wg.Add(1)
go func(queryParams QueryParams, c chan NgxJSON, wg *sync.WaitGroup) {
defer wg.Done()
req, err := http.NewRequest("GET", APIURL2+APIURL2+queryParams.Param2, nil)
if err != nil {
fmt.Println(err)
return
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err) //<-- Error triggers here!!
}
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
var response Response
json.Unmarshal([]byte(string(body)), &response)
//DO THINGS
c <- ngxJSON
}
}(queryParams, c, wg)
go func() {
wg.Wait()
close(c)
}()
//DO THINGS
}
I cant afford the application to terminate.
Then do not panic but handle the 408, e.g. by retrying or returning an error upstream.
Related
I am trying to set up a way to communicate between a central server running Go an a fleet of IoT devices (also running Go).
For each device, it connects to to the central server through a persistent TCPConn. These devices are behind a router(s). The central server saves that connection and sends/receives messages through it. Right now, this is fully functional and works.
However, now the message passing is getting complicated enough that the utility provided by HTTP rather than pure TCP is becoming necessary.
I have attempted to write a version of http.Transport that returns said connection. However, I am unable to provide and return a valid connection from the the Dial/DialContext functions.
IoT Device
func main() {
http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
})
tcpAddr, err := net.ResolveTCPAddr("tcp", "###.###.###.###:8533")
if err != nil {
panic(err)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
panic(err)
}
err = conn.SetKeepAlive(true)
if err != nil {
panic(err)
}
err = conn.SetKeepAlivePeriod(time.Second * 10)
if err != nil {
panic(err)
}
fmt.Println("Listening")
if err := http.ListenAndServe(":8080", nil); err != nil {
panic(err)
}
}
Central Server
func main() {
tcpAddr, err := net.ResolveTCPAddr("tcp", port)
if err != nil {
panic(err)
}
listener, err := net.ListenTCP("tcp", tcpAddr)
if err != nil {
panic(err)
}
conn, err := listener.AcceptTCP()
if err != nil {
panic(err)
}
fmt.Println("Received conn, attempting to send HTTP through connection")
dialFunc := func(network, addr string) (net.Conn, error) {
return conn, nil
}
t := http.Transport{
Dial: dialFunc,
}
client := http.Client{
Transport: &t,
}
fmt.Println("Making request")
res, err := client.Get("http://www.shouldNotMatter.com:8080/foo") // HANGS HERE
if err != nil {
fmt.Println(err)
}
fmt.Println("Received response")
defer res.Body.Close()
if res.StatusCode == http.StatusOK {
bodyBytes, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
bodyString := string(bodyBytes)
fmt.Println(bodyString)
} else {
fmt.Println(res)
}
}
Upon using a debugger to see where it hangs, it seems that it gets stuck in a select statement during the pconn roundtrip. Line 2420 in https://golang.org/src/net/http/transport.go?s=3397:10477
Create a type that returns an existing connection from a dial method:
type connDialer struct {
c net.Conn
}
func (cd connDialer) Dial(network, addr string) (net.Conn, error) {
return cd.c, nil
}
Use the dial method value in a transport:
client := http.Client{Transport: &http.Transport{Dial: connDialer{c}.Dial}}
where c is the existing net.Conn.
Try it on the playground (it works for at one request. It will fail when the client dials a second connection).
The overall approach is fragile. Consider using WebSockets, gRPC or other protocols that are designed to support bi-directional communication.
You missed the code on the client code. The client makes an idle connection to the server and does nothing with it so the connection will definitely hang up. You need to pass the connection to the HTTP server. This can be achieved by using net/http.Serve and passing a net.Listener to it.
type connListener struct {
conn net.Conn
ch chan struct{}
}
func (c connListener) Accept() (Conn, error) {
if c.conn != nil {
conn := c.conn
c.conn = nil
return conn, nil
}
<-c.ch
return nil, errors.New("listener closed")
}
func (c connListener) Close() error {
close(c.ch)
return nil
}
func (c connListener) Addr() net.Addr {
return c.conn.LocalAddr()
}
// call it like this
http.Serve(connListener{conn, make(chan struct{})}, nil)
BTW are you having the client connect to the server and then reverse the connection, making the client behave like an HTTP server and the server behave like the HTTP client? You might want to google "reverse http" for some information on this.
I'm using httputil.ReverseProxy with an http.RoundTripper of my own implementation that uses an ssh.Channel as a transport. My RoundTrip method looks approximately like this:
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
// defer ch.Close()
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
return http.ReadResponse(bufio.NewReader(ch), req)
}
func (c SSHConnection) GetChannel() (ssh.Channel, error) {
ch, req, err := c.Conn.OpenChannel("forwarded-tcpip", msg)
if err != nil {
return nil, err
}
go ssh.DiscardRequests(req)
return ch, nil
}
Notice the commented-out defer ch.Close(). Initially I naively closed the connection here, but the response body would sometimes be empty, due to a race between the HTTP proxy's reading of the body and this closing of the SSH channel.
Assuming, for now, that I don't care to do keep-alive, when can I close the ssh.Channel? If I don't, every request starts a new goroutine (because of go ssh.DiscardRequests(req)), so I leak a goroutine on every HTTP requests until the underlying SSH connection is closed.
An http.RoundTripper should not close the connection until after the response body has been fully consumed, or at the request of the server.
The simplest option is to fully buffer the response and close the connection immediately. In some cases this may actually be the most efficient, if the traffic mostly consists of small, independent requests.
The next option is to hook the closing of the response body to close the channel.
type Body struct {
io.ReadCloser
channel ssh.Channel
}
func (b *Body) Close() error {
b.channel.Close()
return b.ReadCloser.Close()
}
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
resp, err := http.ReadResponse(bufio.NewReader(ch), req)
if err != nil {
ch.Close()
return nil, err
}
resp.Body = &Body{
ReadCloser: resp.Body,
channel: ch,
}
return resp, err
}
Lastly, to make the most efficient use of the ssh channels, you could use an existing Transport with a net.Dialer which makes the ssh connection, and wraps the channel in a net.Conn interface.
Here is the schema :
Client sends a POST request to server A
server A process this and sends a GET to server B
server B sends a response through A to the client
I though the best idea was to make a pipe which would read the response of the GET, and write into the response of the POST, but I got many types problems.
func main() {
r := mux.NewRouter()
r.HandleFunc("/test/{hash}", testHandler)
log.Fatal(http.ListenAndServe(":9095", r))
}
func handleErr(err error) {
if err != nil {
log.Fatalf("%s\n", err)
}
}
func testHandler(w http.ResponseWriter, r *http.Request){
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
write, _ = http.Get("http://localhost:9090/test/" + hash)
defer write.Close()
}()
w.Write(read)
}
When I run this I get the following error:
./ReverseProxy.go:61: cannot use read (type *io.PipeReader) as type []byte in argument to w.Write
Is there a way, to properly insert a io.PipeReader format into an http response?
Or am I doing this in a totally wrong way?
You are not actually writing to it, you're replacing the pipe's write.
Something along the lines of:
func testHandler(w http.ResponseWriter, r *http.Request) {
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
defer write.Close()
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
return
}
defer resp.Body.Close()
io.Copy(write, resp.Body)
}()
io.Copy(w, read)
}
Although, I agree with #JimB, for this instance, the pipe isn't even needed, something like this should be more efficient:
func testHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
hash := vars["hash"]
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
// handle error
return
}
defer resp.Body.Close()
io.Copy(w, resp.Body)
}
I have a server that listens to 2 ports and transfers messages between the 2 ports. The only problem is that each client (or each port) has to submit a message before they can receive the previous message that the other port sent. For example, if client 1 says "hello", client 2 will not receive the message unless he sends a message first.
I want to find a way to make it so that each client will have to receive the previous message before being able to send a new one. (Since Client 1 will not have anything to receive until Client 2 says something, I am planning to send it a string that has placeholder text.)
Can anyone help me with how to go about this? I try putting my listening code and my code for writing a message for each client into their own go routines, but that did not do the job. Any help will be appreciated.
Here is my code:
Server
package main
import (
"fmt"
"log"
"net"
)
var message string = "Client 2 is receiving your message"
func main() {
fmt.Println("The server is listening on Port 3000 and 8080")
//Set up listeners for the ports each client is using
listener, err := net.Listen("tcp", "localhost:3000")
if err != nil {
log.Fatal(err)
}
listener2, err := net.Listen("tcp", "localhost:8080")
if err != nil {
log.Fatal(err)
}
go acceptLoop(listener)
acceptLoop(listener2) // run in the main goroutine
}
func acceptLoop(l net.Listener) {
defer l.Close()
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
fmt.Println("New connection found!")
go listenConnection(c)
}
}
func listenConnection(conn net.Conn) {
fmt.Println("Yay")
for {
buffer := make([]byte, 1400)
dataSize, err := conn.Read(buffer)
if err != nil {
fmt.Println("Connection has closed")
return
}
//This is the message you received
data := buffer[:dataSize]
fmt.Print("Received message: ", string(data))
// Send the message back
_, err = conn.Write([]byte(message))
if err != nil {
log.Fatalln(err)
}
fmt.Print("Message sent: ", string(data))
message = string(data)
}
}
Client 1
package main
import (
"fmt"
"log"
"net"
"bufio"
"os"
)
func main() {
conn, err := net.Dial("tcp", "localhost:3000")
if err != nil {
log.Fatalln(err)
}
for {
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter text: ")
text, _ := reader.ReadString('\n')
_, err = conn.Write([]byte(text))
if err != nil {
log.Fatalln(err)
}
for {
buffer := make([]byte, 1400)
dataSize, err := conn.Read(buffer)
if err != nil {
fmt.Println("The connection has closed!")
return
}
data := buffer[:dataSize]
fmt.Println("Received message: ", string(data))
break
}
}
}
My second client is the same as my first except "localhost:3000" is replaced with "localhost:8080".
I will appreciate any help! I'm fairly new to networking and Go, so any tips will be great.
Right now your Server is just an echo. It responds with the message received from the client.
If you want to relay the message between two clients, the first thing you need to figure out is what should happen if 3 clients connect (2 to the first port, 1 to the other)?
If you expect to have just one client connected to each port you could do something like this (pseudo-go-code):
func main() {
// ...
# Channels must be buffered or else clients will deadlock.
relay := make(chan string, 1)
go acceptLoop(listener1, relay)
acceptLoop(listener2, relay)
}
func acceptLoop(l, relay) {
for {
conn := l.Accept()
for {
relay <- conn.Read()
conn.Write(<-relay)
}
}
I use goroutines achieve http.Get timeout, and then I found that the number has been rising steadily goroutines, and when it reaches 1000 or so, the program will exit
Code:
package main
import (
"errors"
"io/ioutil"
"log"
"net"
"net/http"
"runtime"
"time"
)
// timeout dialler
func timeoutDialler(timeout time.Duration) func(network, addr string) (net.Conn, error) {
return func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
}
func timeoutHttpGet(url string) ([]byte, error) {
// change dialler add timeout support && disable keep-alive
tr := &http.Transport{
Dial: timeoutDialler(3 * time.Second),
DisableKeepAlives: true,
}
client := &http.Client{Transport: tr}
type Response struct {
resp []byte
err error
}
ch := make(chan Response, 0)
defer func() {
close(ch)
ch = nil
}()
go func() {
resp, err := client.Get(url)
if err != nil {
ch <- Response{[]byte{}, err}
return
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
ch <- Response{[]byte{}, err}
return
}
tr.CloseIdleConnections()
ch <- Response{body, err}
}()
select {
case <-time.After(5 * time.Second):
return []byte{}, errors.New("timeout")
case response := <-ch:
return response.resp, response.err
}
}
func handler(w http.ResponseWriter, r *http.Request) {
_, err := timeoutHttpGet("http://google.com")
if err != nil {
log.Println(err)
return
}
}
func main() {
go func() {
for {
log.Println(runtime.NumGoroutine())
time.Sleep(500 * time.Millisecond)
}
}()
s := &http.Server{
Addr: ":8888",
ReadTimeout: 15 * time.Second,
WriteTimeout: 15 * time.Second,
}
http.HandleFunc("/", handler)
log.Fatal(s.ListenAndServe())
}
http://play.golang.org/p/SzGTMMmZkI
Init your chan with 1 instead of 0:
ch := make(chan Response, 1)
And remove the defer block that closes and nils ch.
See: http://blog.golang.org/go-concurrency-patterns-timing-out-and
Here is what I think is happening:
after the 5s timeout, timeoutHttpGet returns
the defer statement runs, closing ch and then setting it to nil
the go routine it started to do the actual fetch finishes and attempts to send its data to ch
but ch is nil, and so won't receive anything, preventing that statement from finishing, and thus preventing the go routine from finishing
I assume you are setting ch = nil because before you had that, you would get run-time panics because that's what happens when you attempt to write to a closed channel, as described by the spec.
Giving ch a buffer of 1 means that the fetch go routine can send to it without needing a receiver. If the handler has returned due to timeout, everything will just get garbage collected later on.