I'm trying to send an HTTP request over a net.Conn TCP connection and read the subsequent response, but I never seem to receive an EOF from the connection when reading. This makes functions like io.Copy or ioutil.ReadAll useless, as they block forever.
Code:
client.go
const requestString = "GET /test HTTP/1.1\r\n" + "Host: 127.0.0.1:8080\r\n\r\n"
func main() {
dialer := net.Dialer{}
conn, err := dialer.Dial("tcp", "127.0.0.1:8080")
if err != nil {
log.Fatalln(err)
}
defer conn.Close()
_, err = conn.Write([]byte(requestString))
if err != nil {
log.Fatalln(err)
}
buf := make([]byte, 1024)
data := make([]byte, 0)
length := 0
for {
n, err := conn.Read(buf)
if err != nil {
if err != io.EOF {
fmt.Printf("Read error: %v\n", err)
}
break
}
data = append(data, buf[:n]...)
length += n
fmt.Printf("Partial read:\n%s\n", string(buf[:n]))
}
fmt.Println("Response:")
fmt.Println(string(data))
}
server.go
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, %s", r.URL.Path[1:])
})
http.ListenAndServe(":8080", nil)
}
Output of running client.go with server.go already running:
Partial read:
HTTP/1.1 200 OK
Date: Wed, 25 Nov 2020 04:09:32 GMT
Content-Length: 11
Content-Type: text/plain; charset=utf-8
Hello, test
The first call to Read() returns the expected response, but lacks an EOF. The subsequent call to Read() hangs forever, and I'm not sure how to determine when the connection has finished. If I interrupt the server process, the client connection closes properly and the response is complete.
How can I either a) receive an EOF or b) determine when the response is complete? All examples I've seen online have something close to my code working, so I'm not sure what I'm doing wrong.
Adding Connection: close to the request headers makes sure that the connection... well, closes.
Related
I am trying to write a simple Unix domain socket based http server in golang. Here is the code:
package main
import (
"fmt"
"log"
"net/http"
"net"
"os"
"encoding/json"
"os/signal"
"syscall"
)
type UnixListener struct {
ServerMux *http.ServeMux
SocketPath string
UID int
GID int
SocketFileMode os.FileMode
}
//////////////////////////////////////////////////////////////////////
// Start the HTTP server with UNIX socket.
//////////////////////////////////////////////////////////////////////
func (l *UnixListener) Start() error {
listener, err := net.Listen("unix", l.SocketPath)
if err != nil {
return err
}
if err := os.Chown(l.SocketPath, l.UID, l.GID); err != nil {
return err
}
if err := os.Chmod(l.SocketPath, l.SocketFileMode); err != nil {
return err
}
go func(){
shutdown(listener)
}()
svr := http.Server{
Handler: l.ServerMux,
}
if err := svr.Serve(listener); err != nil {
return err
} else {
return nil
}
}
//////////////////////////////////////////////////////////////////////
// Shutdown the HTTP server.
//////////////////////////////////////////////////////////////////////
func shutdown(listener net.Listener) {
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
s := <-c
listener.Close()
log.Fatalf("[FATAL] Caught a signal. sinal=%s", s)
}
func hello(w http.ResponseWriter, r *http.Request) {
log.Println("Received request %s", r.RequestURI)
fmt.Fprintf(w, "Hello World!\n")
w.Header().Set("ContentType", "application/json")
w.WriteHeader(http.StatusOK)
data := map[string]string { "name" : "satish"}
resp, _ := json.Marshal(data)
if _, err := w.Write(resp); err != nil {
fmt.Printf("Error writing response")
}
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/hello", hello)
unixListener := UnixListener{
ServerMux: mux,
SocketPath: "/temp/test.sock",
UID: 501,
GID: 20,
SocketFileMode: 0644,
}
if err := unixListener.Start(); err != nil {
log.Fatalf("[FATAL] HTTP server error: %s", err)
}
}
But when I send the request to this UDS based server like below, the handler does not seem to be invoked, giving 404 not found error. What is the right way to implement UDS based http server and how to send the http request to it?
curl -vvv --unix-socket /temp/test.sock http:/hello
* Trying /Users/sburnwal/Projects/ACI/temp/test.sock:0...
* Connected to hello (/temp/test.sock) port 80 (#0)
> GET / HTTP/1.1
> Host: hello
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Mon, 14 Nov 2022 02:38:11 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host hello left intact
It appears it's interpreting http:/hello in your command line as the host name and using that for the header. And then requesting / as the path
Try replacing http:/hello in your command line with http://hello/hello. I'm not sure if the single slash vs double-slash is significant. It just looks suspicious.
curl -vvv --unix-socket /temp/test.sock http://hello/hello
Or change your Go code to have a response handler for "/" instead of just for "/hello"
mux.HandleFunc("/hello", hello)
mux.HandleFunc("/", hello)
When throwing an HTTP Request with Go and receiving a Response, I want to receive a response while streaming, considering the case where the ResponseBody is huge (1 GB or more).
resp, err: = http.Client.Do(req)
In this case, if the body is huge, I can not read the Header and I do not know the state of Response.
Is there any solution?
(Edit: If you're unable to get the "Content-length" header from the response, it is possible that the web service you're hitting doesn't return that header. In such a case, there's no way to know the length of the response body without reading it completely. You can simulate that in the following example by removing the line that sets the Content-length header in the response.)
The standard Go net/http package handles large responses very well. Here's a self contained example to demonstrate:
// Start a mock HTTP server that returns 2GB of data in the response. Make a
// HTTP request to this server and print the amount of data read from the
// response.
package main
import (
"fmt"
"io"
"log"
"net/http"
"strings"
"time"
)
const oneMB = 1024 * 1024
const oneGB = 1024 * oneMB
const responseSize = 2 * oneGB
const serverAddr = "localhost:9999"
func startServer() {
// Mock HTTP server that always returns 2GB of data
go http.ListenAndServe(serverAddr, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-length", fmt.Sprintf("%d", responseSize))
// 1MB buffer that'll be copied multiple times to the response
buf := []byte(strings.Repeat("x", oneMB))
for i := 0; i < responseSize/len(buf); i++ {
if _, err := w.Write(buf); err != nil {
log.Fatal("Failed to write to response. Error: ", err.Error())
}
}
}))
// Some grace period for the server to start
time.Sleep(100 * time.Millisecond)
}
func main() {
startServer()
// HTTP client
req, err := http.NewRequest("GET", "http://"+serverAddr, nil)
if err != nil {
log.Fatal("Error creating HTTP request: ", err.Error())
}
client := http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal("Error making HTTP request: ", err.Error())
}
// Read the response header
fmt.Println("Response: Content-length:", resp.Header.Get("Content-length"))
bytesRead := 0
buf := make([]byte, oneMB)
// Read the response body
for {
n, err := resp.Body.Read(buf)
bytesRead += n
if err == io.EOF {
break
}
if err != nil {
log.Fatal("Error reading HTTP response: ", err.Error())
}
}
fmt.Println("Response: Read", bytesRead, "bytes")
}
You wouldn't want to read the entire response in memory if it's too large. Write it to a temporary file instead and then process that.
If instead you're looking for options to do this reliably when the network isn't very reliable, look for "HTTP range requests" using which you can resume partially downloaded data.
I'm using httputil.ReverseProxy with an http.RoundTripper of my own implementation that uses an ssh.Channel as a transport. My RoundTrip method looks approximately like this:
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
// defer ch.Close()
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
return http.ReadResponse(bufio.NewReader(ch), req)
}
func (c SSHConnection) GetChannel() (ssh.Channel, error) {
ch, req, err := c.Conn.OpenChannel("forwarded-tcpip", msg)
if err != nil {
return nil, err
}
go ssh.DiscardRequests(req)
return ch, nil
}
Notice the commented-out defer ch.Close(). Initially I naively closed the connection here, but the response body would sometimes be empty, due to a race between the HTTP proxy's reading of the body and this closing of the SSH channel.
Assuming, for now, that I don't care to do keep-alive, when can I close the ssh.Channel? If I don't, every request starts a new goroutine (because of go ssh.DiscardRequests(req)), so I leak a goroutine on every HTTP requests until the underlying SSH connection is closed.
An http.RoundTripper should not close the connection until after the response body has been fully consumed, or at the request of the server.
The simplest option is to fully buffer the response and close the connection immediately. In some cases this may actually be the most efficient, if the traffic mostly consists of small, independent requests.
The next option is to hook the closing of the response body to close the channel.
type Body struct {
io.ReadCloser
channel ssh.Channel
}
func (b *Body) Close() error {
b.channel.Close()
return b.ReadCloser.Close()
}
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
resp, err := http.ReadResponse(bufio.NewReader(ch), req)
if err != nil {
ch.Close()
return nil, err
}
resp.Body = &Body{
ReadCloser: resp.Body,
channel: ch,
}
return resp, err
}
Lastly, to make the most efficient use of the ssh channels, you could use an existing Transport with a net.Dialer which makes the ssh connection, and wraps the channel in a net.Conn interface.
This is actually a follow up on this question: How to know TCP connection is closed in Golang net package? which doesn't work in my implementation, and I'm trying to figure out why.
The idea is, I have a server that accepts TCP connections and waits for the client to close them. It waits for 5 seconds, and every second it tries to read from the TCP connection, if this gives an EOF error, that would indicate that the client has closed the connection.
This is the servers code:
func server(done chan bool) {
l, _ := net.Listen("tcp", serverAddress)
// Listen for incomming TCP connections
conn, err := l.Accept()
if err != nil {
log.Fatalf("l.Accept(): got error on accept: %v", err)
}
defer conn.Close()
// Read periodically from the TCP connection to see if we can find
// out when the client has closed his end.
go func() {
for {
conn.SetReadDeadline(time.Now().Add(10 * time.Millisecond))
b, err := conn.Read([]byte{})
fmt.Printf("Read %v bytes with err %v \n", b, err)
if err == io.EOF {
conn.Close()
conn = nil
break
} else {
var zero time.Time
conn.SetReadDeadline(zero)
}
time.Sleep(1 * time.Second)
}
done <- true
}()
// Do this for 5 seconds, than quit
time.Sleep(5 * time.Second)
done <- true
}
So the client sets up a connection, waits for two seconds and then closes it. The clients code is as follows:
func main() {
done := make(chan bool, 1)
go server(done)
// Give the listener some time to set up
time.Sleep(200 * time.Millisecond)
// Dial the server
conn, err := net.Dial("tcp", serverAddress)
if err != nil {
log.Fatalf("net.Dial(): got %v, want no error", err)
}
defer conn.Close()
go func() {
// After two seconds, close the TCP connection to the server
time.Sleep(2 * time.Second)
fmt.Printf("Client closes conn now with err %v\n", conn.Close())
}()
// Wait untill server is done
<-done
}
I would expect that the server would get an EOF error when reading, after the client has closed the connection, however the output I get is:
Read 0 bytes with err <nil>
Read 0 bytes with err <nil>
Client closes conn now with err <nil>
Read 0 bytes with err <nil>
Read 0 bytes with err <nil>
Read 0 bytes with err <nil>
I have the wireshark output here. It actually shows the FIN was send by the client (package 11), but only 3 seconds later the server sends it's own FIN:
Wireshark output
So basically I can't figure out why I'm not getting any error on the read on the servers side.
i use
resp, err := http.Get("http://example.com/")
get a http.Response, and i want to exactly write to a http handler, but only http.ResponseWriter, so i hijack it.
...
webConn, webBuf, err := hj.Hijack()
if err != nil {
// handle error
}
defer webConn.Close()
// Write resp
resp.Write(webBuf)
...
Write raw request
But When i hijack, http connection can't reuse (keep-alive), so it slow.
How to solve?
Thanks! Sorry for my pool English.
update 12/9
keep-alive, It keep two tcp connection, and can reuse.
but when i hijack, and conn.Close(), It can't reuse old connection, so it create a new tcp connection when i each refresh.
Do not use hijack, Because once hijack, the HTTP server library will not do anything else with the connection, So can't reuse.
I change way, copy Header and Body, look like reverse proxy (http://golang.org/src/pkg/net/http/httputil/reverseproxy.go), Is works.
Example:
func copyHeader(dst, src http.Header) {
for k, w := range src {
for _, v := range w {
dst.Add(k, v)
}
}
}
func copyResponse(r *http.Response, w http.ResponseWriter) {
copyHeader(w.Header(), r.Header)
w.WriteHeader(r.StatusCode)
io.Copy(w, r.Body)
}
func handler(w http.ResponseWriter, r *http.Response) {
resp, err := http.Get("http://www.example.com")
if err != nil {
// handle error
}
copyResponse(resp, w)
}
It seem that once the connection is closed the keep-alive connection closes as well.
One possible solution would be to prevent the connection from closing until desired, but I'm not sure if that good advise.
Maybe the correct solution involves creating a instance of net.TCPConn, copying the connection over it, then calling .SetKeepAlive(true).
Before running the below example, launch another terminal with netstat -antc | grep 9090.
Routes in example:
localhost:9090/ok is a basic (non-hijacked) connection
localhost:9090 is a hijacked connection, lasting for 10 seconds.
Example
package main
import (
"fmt"
"net/http"
"sync"
"time"
)
func checkError(e error) {
if e != nil {
panic(e)
}
}
var ka_seconds = 10
var conn_id = 0
func main() {
http.HandleFunc("/ok", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "ok")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
conn_id++
fmt.Printf("Connection %v: Keep-alive is enabled %v seconds\n", conn_id, ka_seconds)
hj, ok := w.(http.Hijacker)
if !ok {
http.Error(w, "webserver doesn't support hijacking", http.StatusInternalServerError)
return
}
conn, bufrw, err := hj.Hijack()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Don't forget to close the connection:
time.AfterFunc(time.Second* time.Duration(ka_seconds), func() {
conn.Close()
fmt.Printf("Connection %v: Keep-alive is disabled.\n", conn_id)
})
resp, err := http.Get("http://www.example.com")
checkError(err)
resp.Write(bufrw)
bufrw.Flush()
})
fmt.Println("Listing to localhost:9090")
http.ListenAndServe(":9090", nil)
}
Related issue: http://code.google.com/p/go/issues/detail?id=5645