http://play.golang.org/p/gZo5RqgY4F
I have a question with io.Copy method. The link above will block in line 44 under Go 1.5. But will pass in 1.4.2. I have no idea with this issue.
Here is my go version: go version go 1.5 darwin/amd64.
When did the io.Copy return in go 1.5?
Previously, you were getting lucky when the timing of the syscalls would cause a write error in your second io.Copy [line 41]. (Ignoring errors tends to hide bugs)
This was purely by accident (and may even be incorrect). Since the source connection of that copy (conn2) is never closed, the io.Copy never receives an io.EOF and doesn't return. You need to close the opposing connection in each of the copy goroutines to unblock the other's call to io.Copy.
wg.Add(1)
go func() {
io.Copy(conn1, conn2)
// conn2 has returned EOF or an error, so we need to shut down the
// other half of the duplex copy.
conn1.Close()
wg.Done()
}()
wg.Add(1)
go func() {
io.Copy(conn2, conn1)
conn2.Close()
wg.Done()
}()
Related
func serveApp() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(resp http.ResponseWriter, req *http.Request) {
fmt.Fprintln(resp, "Hello, QCon!")
})
http.ListenAndServe("0.0.0.0:8080", mux)
}
func serveDebug() {
http.ListenAndServe("127.0.0.1:8001", http.DefaultServeMux)
}
func main() {
go serveDebug()
serveApp()
}
However, serveDebug is run in a separate goroutine and if it returns
just that goroutine will exit while the rest of the program continues
on. Your operations staff will not be happy to find that they cannot
get the statistics out of your application when they want too because
the /debug handler stopped working a long time ago.
I am new to Golang and coding in general. I came across an article online, and found this code. I copy and paste it into my editor and type go run main.go. The program runs forever without any errors. I can curl it with no problems. Why is it bad code? I am a noob, and I am trying to get a better understanding of this, if this could be explained in simple terms that would be great.
The program creates two HTTP servers to respond to traffic received on to different ports. The debug server is run in a separate goroutine, and there is no way to detect if that server failed. The program may continue running serving with the App server.
A better implementation would be to stop both servers if one of them failed:
stop:=make(chan struct{},2)
go func() {
defer func() {
stop<-struct{}{}
}()
serveDebug()
}()
go func() {
defer func() {
stop <-struct{}{}
}{}
serveApp()
}()
<-stop
Above, the program will create two goroutines and block at <-stop until someone writes to the channel. If any one of the servers fail, the goroutine will write to the channel, which will unblock <-stop, so the program will exit.
When I look at the net/http server interface, I don't see an obvious way to get notified and react when the http.Server comes up and starts listening:
ListenAndServe(":8080", nil)
The function doesn't return until the server actually shuts down. I also looked at the Server type, but there doesn't appear to be anything that lets me tap into that timing. Some function or a channel would have been great but I don't see any.
Is there any way that will let me detect that event, or am I left to just sleeping "enough" to fake it?
ListenAndServe is a helper function that opens a listening socket and then serves connections on that socket. Write the code directly in your application to signal when the socket is open:
l, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
// Signal that server is open for business.
if err := http.Serve(l, rootHandler); err != nil {
// handle error
}
If the signalling step does not block, then http.Serve will easily consume any backlog on the listening socket.
Related question: https://stackoverflow.com/a/32742904/5728991
I am building a tool in Go that needs to make a very large number of simultaneous HTTP requests to many different servers. My initial prototype in Python had no problem doing a few hundred simultaneous requests.
However, I have found that in Go this almost always results in a Get http://www.google.com: dial tcp 216.58.205.228:80: i/o timeout for some if the number of simultaneous requests exceeds ~30-40.
I've tested on macOS, openSUSE, different hardware, in different networks and with different domain lists, and changing the DNS server as described in other Stackoverflow answers does not work either.
The interesting thing is that the failed requests do not even produce a packet, as can be seen when checking with Wireshark.
Is there anything that I am doing wrong or is that a bug in Go?
Minimum reproducible program below:
package main
import (
"fmt"
"net/http"
"sync"
)
func main() {
domains := []string{/* large domain list here, eg from https://moz.com/top500 */}
limiter := make(chan string, 50) // Limits simultaneous requests
wg := sync.WaitGroup{} // Needed to not prematurely exit before all requests have been finished
for i, domain := range domains {
wg.Add(1)
limiter <- domain
go func(i int, domain string) {
defer func() { <-limiter }()
defer wg.Done()
resp, err := http.Get("http://"+domain)
if err != nil {
fmt.Printf("%d %s failed: %s\n", i, domain, err)
return
}
fmt.Printf("%d %s: %s\n", i, domain, resp.Status)
}(i, domain)
}
wg.Wait()
}
Two particular error messages are happening, a net.DNSError that does not make any sense and a non-descript poll.TimeoutError:
&url.Error{Op:"Get", URL:"http://harvard.edu", Err:(*net.OpError)(0xc00022a460)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc000aca200)}
&net.DNSError{Err:"no such host", Name:"harvard.edu", Server:"", IsTimeout:false, IsTemporary:false}
&url.Error{Op:"Get", URL:"http://latimes.com", Err:(*net.OpError)(0xc000d92730)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*poll.TimeoutError)(0x14779a0)}
&poll.TimeoutError{}
Update:
Running the requests with a seperate http.Client as well as http.Transport and net.Dialer does not make any difference as can be seen when running code from this playground.
I think many of your net.DNSErrors are actually too many open files errors in disguise. You can see this by running your sample code with the netgo tag (recommendation from here) (go run -tags netgo main.go) which will emit errors like:
…dial tcp: lookup buzzfeed.com on 192.168.1.1:53: dial udp 192.168.1.1:53: socket: too many open files
instead of
…dial tcp: lookup buzzfeed.com: no such host
Make sure you're closing the request's response body (resp.Body.Close()). You can find more about this specific problem at What's the best way to handle "too many open files"? and How to set ulimit -n from a golang program?. (On my machine (macOS), increasing file limits manually seemed to help, but I don't think it's a good solution since it doesn't really scale, and I'm not sure how many open files you'd need overall.)
As suggested by #liam-kelly, I think the i/o timeout error is coming from a DNS server or some other security mechanism. Setting a custom (bad) DNS server IP gives me the same error.
On the client side of a TCP connection, I am attempting to to reuse established connections as much as possible to avoid the overhead of dialing every time I need a connection. Fundamentally, it's connection pooling, although technically, my pool size just happens to be one.
I'm running into a problem in that if a connection sits idle for long enough, the other end disconnects. I've tried using something like the following to keep connections alive:
err = conn.(*net.TCPConn).SetKeepAlive(true)
if err != nil {
fmt.Println(err)
return
}
err = conn.(*net.TCPConn).SetKeepAlivePeriod(30*time.Second)
if err != nil {
fmt.Println(err)
return
}
But this isn't helping. In fact, it's causing my connections to close sooner. I'm pretty sure this is because (on a Mac) this means the connection health starts being probed after 30 seconds and then is probed at 8 times at 30 second intervals. The server side must not be supporting keepalive, so after 4 minutes and 30 seconds, the client is disconnecting.
There might be nothing I can do to keep an idle connection alive indefinitely, and that would be absolutely ok if there were some way for me to at least detect that a connection has been closed so that I can seamlessly replace it with a new one. Alas, even after reading all the docs and scouring the blogosphere for help, I can't find any way at all in go to query the state of a TCP connection.
There must be a way. Does anyone have any insight into how that can be accomplished? Many thanks in advance to anyone who does!
EDIT:
Ideally, I'd like to learn how to handle this, low-level with pure go-- without using third-party libraries to accomplish this. Of course if there is some library that does this, I don't mind being pointed in its direction so I can see how they do it.
The socket api doesn't give you access to the state of the connection. You can query the current state it in various ways from the kernel (/proc/net/tcp[6] on linux for example), but that doesn't make any guarantee that further sends will succeed.
I'm a little confused on one point here. My client is ONLY sending data. Apart from acking the packets, the server sends nothing back. Reading doesn't seem an appropriate way to determine connection status, as there's noting TO read.
The socket API is defined such that that you detect a closed connection by a read returning 0 bytes. That's the way it works. In Go, this is translated to a Read returning io.EOF. This will usually be the fastest way to detect a broken connection.
So am I supposed to just send and act on whatever errors occur? If so, that's a problem because I'm observing that I typically do not get any errors at all when attempting to send over a broken pipe-- which seems totally wrong
If you look closely at how TCP works, this is the expected behavior. If the connection is closed on the remote side, then your first send will trigger an RST from the server, fully closing the local connection. You either need to read from the connection to detect the close, or if you try to send again you will get an error (assuming you've waited long enough for the packets to make a round trip), like "broken pipe" on linux.
To clarify... I can dial, unplug an ethernet cable, and STILL send without error. The messages don't get through, obviously, but I receive no error
If the connection is actually broken, or the server is totally unresponsive, then you're sending packets off to nowhere. The TCP stack can't tell the difference between packets that are really slow, packet loss, congestion, or a broken connection. The system needs to wait for the retransmission timeout, and retry the packet a number of times before failing. The standard configuration for retries alone can take between 13 and 30 minutes to trigger an error.
What you can do in your code is
Turn on keepalive. This will notify you of a broken connection more quickly, because the idle connection is always being tested.
Read from the socket. Either have a concurrent Read in progress, or check for something to read first with select/poll/epoll (Go usually uses the first)
Set timeouts (deadlines in Go) for everything.
If you're not expecting any data from the connection, checking for a closed connection is very easy in Go; dispatch a goroutine to read from the connection until there's an error.
notify := make(chan error)
go func() {
buf := make([]byte, 1024)
for {
n, err := conn.Read(buf)
if err != nil {
notify <- err
return
}
if n > 0 {
fmt.Println("unexpected data: %s", buf[:n])
}
}
}()
There is no such thing as 'TCP connection state', by design. There is only what happens when you send something. There is no TCP API, at any level down to the silicon, that will tell you the current state of a TCP connection. You have to try to use it.
If you're sending keepalive probes, the server doesn't have any choice but to respond appropriately. The server doesn't even know that they are keepalives. They aren't. They are just duplicate ACKs. Supporting keepalive just means supporting sending keepalives.
I have a simple golang program which listens for activity on a port before executing a function called testFunc
func main() {
http.HandleFunc("/test", testFunc)
http.ListenAndServe(":1337", nil)
}
When I build this program, run it and go to http://localhost:1337/test, it works fine.
When I terminate the program and try and run it again, the program instantly terminates showing no error output.
When I then change the port to 1338 for example, it works the first time, then fails each time after. Any ideas?
The port is most likely in use. Catching the error will give you more details.
if err := http.ListenAndServe(":1337", nil);err != nil {
log.Fatal("ListenAndServe: ", err)
}
Ok turns out the error was my silly mistake. I have today switched from using a PC to Mac and didn't realise that Ctrl+C killed the current command instead of Ctrl+z which simply returns to shell... Therefore the processes were still running and blocking each other