Golang TCP connection close doesn't give an EOF - networking

Question

This is actually a follow up on this question: How to know TCP connection is closed in Golang net package? which doesn't work in my implementation, and I'm trying to figure out why.
The idea is, I have a server that accepts TCP connections and waits for the client to close them. It waits for 5 seconds, and every second it tries to read from the TCP connection, if this gives an EOF error, that would indicate that the client has closed the connection.
This is the servers code:
func server(done chan bool) {
l, _ := net.Listen("tcp", serverAddress)
// Listen for incomming TCP connections
conn, err := l.Accept()
if err != nil {
log.Fatalf("l.Accept(): got error on accept: %v", err)
}
defer conn.Close()
// Read periodically from the TCP connection to see if we can find
// out when the client has closed his end.
go func() {
for {
conn.SetReadDeadline(time.Now().Add(10 * time.Millisecond))
b, err := conn.Read([]byte{})
fmt.Printf("Read %v bytes with err %v \n", b, err)
if err == io.EOF {
conn.Close()
conn = nil
break
} else {
var zero time.Time
conn.SetReadDeadline(zero)
}
time.Sleep(1 * time.Second)
}
done <- true
}()
// Do this for 5 seconds, than quit
time.Sleep(5 * time.Second)
done <- true
}
So the client sets up a connection, waits for two seconds and then closes it. The clients code is as follows:
func main() {
done := make(chan bool, 1)
go server(done)
// Give the listener some time to set up
time.Sleep(200 * time.Millisecond)
// Dial the server
conn, err := net.Dial("tcp", serverAddress)
if err != nil {
log.Fatalf("net.Dial(): got %v, want no error", err)
}
defer conn.Close()
go func() {
// After two seconds, close the TCP connection to the server
time.Sleep(2 * time.Second)
fmt.Printf("Client closes conn now with err %v\n", conn.Close())
}()
// Wait untill server is done
<-done
}
I would expect that the server would get an EOF error when reading, after the client has closed the connection, however the output I get is:
Read 0 bytes with err <nil>
Read 0 bytes with err <nil>
Client closes conn now with err <nil>
Read 0 bytes with err <nil>
Read 0 bytes with err <nil>
Read 0 bytes with err <nil>
I have the wireshark output here. It actually shows the FIN was send by the client (package 11), but only 3 seconds later the server sends it's own FIN:
Wireshark output
So basically I can't figure out why I'm not getting any error on the read on the servers side.