trying to send more than one of the same HTTP request generated by http.ReadRequest, I'm getting the following error:
Post http://192.168.x.x:8000/dir1: http: invalid Read on closed Body
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5ec4cc]
My code takes a file, and puts it in a bufio.NewReader. It then uses http.ReadRequest on what it had read, which is a HTTP POST request. finally - it opens another file, "wordlist.txt", which contains directories - and iterates over the original HTTP URI each time with a different directory from the list.
For some reason - the second requests causes to program to error out (http: invalid Read on closed Body). Heres a working example (just change the paths to yours):
main.go
package main
import (
"bufio"
"fmt"
"log"
"net/http"
"net/url"
"os"
)
func main() {
//Read HTTP request from text file
rdr, err := os.Open("/opt/gohttp/tests/req2.txt")
if err != nil {
log.Println(err)
}
//Parse to a web request to work with using ReadRequest
req, err := http.ReadRequest(bufio.NewReader(rdr))
if err != nil {
log.Println(err)
}
// fix for "requestURI can't be sent in client requests"
req.RequestURI = ""
//Open wordlist
file, err := os.Open("/opt/gohttp/tests/wordlist.txt")
if err != nil {
log.Println(err)
}
defer file.Close()
//Scan wordlist line by line and append each to the directory
scanner := bufio.NewScanner(file)
for scanner.Scan() {
var i string = scanner.Text()
fmt.Println(i)
//this part appends the wordlist word to the request sent e.g. /dir1
u, err := url.Parse("http://" + req.Host + "/" + i) //todo: https? other protocols?
if err != nil {
log.Println("scanner parse error: ", err)
}
req.URL = u
//Send the request
fmt.Println("Sending request.." + "\n")
client := &http.Client{}
response, err := client.Do(req)
if err != nil {
log.Println(err) // invalid read on closed body error
}
fmt.Println(*response)
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
}
example HTTP request file
POST /hi HTTP/1.1
Host: 192.168.x.x:8000
Content-Length: 41
{"email":"aa","password":"secret"}
example wordlist file
0
1
dir1
dir2
dir3
application output
go run tests/main.go
0
Sending request..
{404 Not Found 404 HTTP/1.1 1 1...}
1
Sending request..
2019/01/28 04:27:52 Post http://192.168.x.x:8000/1: http: invalid Read on closed Body
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5ec4cc]
goroutine 1 [running]:
main.main()
/opt/gohttp/tests/main.go:53 +0x35c
exit status 2
Please note that the first HTTP request is sent successfuly, and only the second one returns the error above. How can I fix this?
Related
I'm trying to send an HTTP request over a net.Conn TCP connection and read the subsequent response, but I never seem to receive an EOF from the connection when reading. This makes functions like io.Copy or ioutil.ReadAll useless, as they block forever.
Code:
client.go
const requestString = "GET /test HTTP/1.1\r\n" + "Host: 127.0.0.1:8080\r\n\r\n"
func main() {
dialer := net.Dialer{}
conn, err := dialer.Dial("tcp", "127.0.0.1:8080")
if err != nil {
log.Fatalln(err)
}
defer conn.Close()
_, err = conn.Write([]byte(requestString))
if err != nil {
log.Fatalln(err)
}
buf := make([]byte, 1024)
data := make([]byte, 0)
length := 0
for {
n, err := conn.Read(buf)
if err != nil {
if err != io.EOF {
fmt.Printf("Read error: %v\n", err)
}
break
}
data = append(data, buf[:n]...)
length += n
fmt.Printf("Partial read:\n%s\n", string(buf[:n]))
}
fmt.Println("Response:")
fmt.Println(string(data))
}
server.go
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, %s", r.URL.Path[1:])
})
http.ListenAndServe(":8080", nil)
}
Output of running client.go with server.go already running:
Partial read:
HTTP/1.1 200 OK
Date: Wed, 25 Nov 2020 04:09:32 GMT
Content-Length: 11
Content-Type: text/plain; charset=utf-8
Hello, test
The first call to Read() returns the expected response, but lacks an EOF. The subsequent call to Read() hangs forever, and I'm not sure how to determine when the connection has finished. If I interrupt the server process, the client connection closes properly and the response is complete.
How can I either a) receive an EOF or b) determine when the response is complete? All examples I've seen online have something close to my code working, so I'm not sure what I'm doing wrong.
Adding Connection: close to the request headers makes sure that the connection... well, closes.
I observed that Content-Length header is not getting set for PATCH requests with empty/nil payload. Even if we manually set it by req.Header.Set("content-length", "0") it is not actually getting set in the out going request.
This strange behaviour (Go bug?) happens only for PATCH requests and only when the payload is empty or nil (or set to http.NoBody)
package main
import (
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
url := "http://localhost:9999"
method := "PATCH"
payload := strings.NewReader("")
client := &http.Client {
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Authorization", "Bearer my-token")
req.Header.Set("Content-Length", "0") //this is not honoured
res, err := client.Do(req)
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
fmt.Println(string(body))
}
This is reproducible even in the latest go version 1.15.
Just run the above code against a simple http server and see for yourself.
Is there any solution/workaround to send a PATCH request with Content-Length set to 0 ?
You can tell the HTTP client to include a Content-Length header with value 0 by setting TransferEncoding to identity as follows:
url := "http://localhost:9999"
method := "PATCH"
client := &http.Client{}
req, err := http.NewRequest(method, url, http.NoBody)
if err != nil {
panic(err)
}
req.TransferEncoding = []string{"identity"}
req.Header.Set("Authorization", "Bearer my-token")
// req.Header.Set("Content-Length", "0")
Note the following changes to your original code:
the important one: req.TransferEncoding = []string{"identity"}
the idiomatic way of specifying an empty body: http.NoBody (no impact on sending the length)
commented out req.Header.Set("Content-Length", "0"), the client fills it in by itself
also changed to panic on an error, you probably don't want to continue
The transfer encoding of identity is not written to the request, so except for the header Content-Length = 0, the request looks the same as before.
This is unfortunately not documented (feel free to file an issue with the Go team), but can be seen in the following code:
The tedious details:
transferWriter.writeHeader checks the following to write the Content-Length header:
// Write Content-Length and/or Transfer-Encoding whose values are a
// function of the sanitized field triple (Body, ContentLength,
// TransferEncoding)
if t.shouldSendContentLength() {
if _, err := io.WriteString(w, "Content-Length: "); err != nil {
return err
}
if _, err := io.WriteString(w, strconv.FormatInt(t.ContentLength, 10)+"\r\n"); err != nil {
return err
}
In turn, shouldCheckContentLength looks at the transfer encoding in case of zero length:
if t.ContentLength == 0 && isIdentity(t.TransferEncoding) {
if t.Method == "GET" || t.Method == "HEAD" {
return false
}
return true
}
The isIdentity verifies that TransferEncoding is exactly []string{"identity"}:
func isIdentity(te []string) bool { return len(te) == 1 && te[0] == "identity" })
I'm perpetually caught out by reading an io.ReadCloser and then forgetting that I've read it before, and when I read it again, I get an empty payload. I wish there was some lint check for my stupidity. Nonetheless, I think I can use TeeReader, but it fails to meet my expectations here:
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
buf := &bytes.Buffer{}
tee := io.TeeReader(r.Body, buf)
body, err := ioutil.ReadAll(tee)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
log.Println("body", string(body))
payload, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("dump request", string(payload))
w.WriteHeader(http.StatusOK)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
The body is missing from my "dump request" log line.
I.e. when I run curl -i -X POST --data '{"username":"xyz","password":"xyz"}' http://localhost:8080
I want the original request in full:
2019/01/14 11:09:50 dump request POST / HTTP/1.1
Host: localhost:8080
Accept: */*
Content-Length: 35
Content-Type: application/x-www-form-urlencoded
User-Agent: curl/7.63.0
{"username":"xyz","password":"xyz"}
What am I missing?
You cannot rewind an io.ReadCloser, unless the underlying value is also an io.ReadSeeker.
An io.ReadCloser, by very definition, has exactly two methods: Read and Close. So there is obviously there is no option to rewind.
An io.ReadSeeker, by contrast, has two methods: Read and Seek, the latter which allows rewinding.
If you need to accept only io.ReadClosers which are also seekable, you can easily combine these two:
type ReadSeekCloser interface {
io.Reader
io.Seeker
io.Closer
}
Now you can use your custom ReadSeekCloser type in place of an io.ReadCloser, and you'll have the option to rewind your reader.
Of course, few io.ReadClosers in the wild actually conform to this interface (os.File will be the main one that does). If you have an io.ReadCloser that does not implement the Seek method (such as a network stream), probably the easiest way to make it Seekable is to dump the contents to a file, then open that file. There would be other ways to make an in-memory buffer seekable (bytes.NewReader for instance), but variations will require reading the stream into memory or onto disk, first.
How do I rewind a io.ReadCloser in Go[...]?
You cannot. A ReadCloser can be read and closed. Unless the actual underlying type has some method to rewind you simply cannot.
(For your case you may just use the bytes.Buffer, possibly after adding a Close method via io/ioutil.ReadCloser as the Request.Body; but this is not "rewinding" but "replacing".)
https://golang.org/pkg/net/http/httputil/#DumpRequest
DumpRequest returns the given request in its HTTP/1.x wire representation. It should only be used by servers to debug client requests.
Clearly DumpRequest is for Dubug use.
But if you don't care about that. The godoc also metioned:
If body is true, DumpRequest also returns the body. To do so, it consumes req.Body and then replaces it with a new io.ReadCloser that yields the same bytes.
So you can call DumpRequest first then ReadAll from Body, cause the Body is still the same after DumpRequest.
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
payload, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("dump request", string(payload))
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
log.Println("body", string(body))
w.WriteHeader(http.StatusOK)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
Request bodies aren't seek-able. There is no rewind, they aren't buffered in memory. When that data is read, it is read from the network stream. Imagine a large upload, buffering all of that data in memory by default would be wasteful.
That said, there are some things you can do to get the output you desire. It involves replacing r.Body after you've read it.
You mention wishing for a lint check. I've found that declaring a buffer and using only that buffer can help. If you like, you can even replace r.Body with that buffer. You will still need to remember to "rewind" with Seek.
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
b := &bytes.Buffer{}
_, err := io.Copy(b, r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
bodyCopy := bytes.NewReader(b.Bytes())
log.Println("body", b.String())
r.Body = io.NopCloser(bodyCopy)
bodyCopy.Seek(0, 0)
payload, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("dump request", string(payload))
w.WriteHeader(http.StatusOK)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
When throwing an HTTP Request with Go and receiving a Response, I want to receive a response while streaming, considering the case where the ResponseBody is huge (1 GB or more).
resp, err: = http.Client.Do(req)
In this case, if the body is huge, I can not read the Header and I do not know the state of Response.
Is there any solution?
(Edit: If you're unable to get the "Content-length" header from the response, it is possible that the web service you're hitting doesn't return that header. In such a case, there's no way to know the length of the response body without reading it completely. You can simulate that in the following example by removing the line that sets the Content-length header in the response.)
The standard Go net/http package handles large responses very well. Here's a self contained example to demonstrate:
// Start a mock HTTP server that returns 2GB of data in the response. Make a
// HTTP request to this server and print the amount of data read from the
// response.
package main
import (
"fmt"
"io"
"log"
"net/http"
"strings"
"time"
)
const oneMB = 1024 * 1024
const oneGB = 1024 * oneMB
const responseSize = 2 * oneGB
const serverAddr = "localhost:9999"
func startServer() {
// Mock HTTP server that always returns 2GB of data
go http.ListenAndServe(serverAddr, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-length", fmt.Sprintf("%d", responseSize))
// 1MB buffer that'll be copied multiple times to the response
buf := []byte(strings.Repeat("x", oneMB))
for i := 0; i < responseSize/len(buf); i++ {
if _, err := w.Write(buf); err != nil {
log.Fatal("Failed to write to response. Error: ", err.Error())
}
}
}))
// Some grace period for the server to start
time.Sleep(100 * time.Millisecond)
}
func main() {
startServer()
// HTTP client
req, err := http.NewRequest("GET", "http://"+serverAddr, nil)
if err != nil {
log.Fatal("Error creating HTTP request: ", err.Error())
}
client := http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal("Error making HTTP request: ", err.Error())
}
// Read the response header
fmt.Println("Response: Content-length:", resp.Header.Get("Content-length"))
bytesRead := 0
buf := make([]byte, oneMB)
// Read the response body
for {
n, err := resp.Body.Read(buf)
bytesRead += n
if err == io.EOF {
break
}
if err != nil {
log.Fatal("Error reading HTTP response: ", err.Error())
}
}
fmt.Println("Response: Read", bytesRead, "bytes")
}
You wouldn't want to read the entire response in memory if it's too large. Write it to a temporary file instead and then process that.
If instead you're looking for options to do this reliably when the network isn't very reliable, look for "HTTP range requests" using which you can resume partially downloaded data.
I am trying to learn the net package. I am listening for local connections on a port, and sending data to that port using echo -n "Server test.\n" | nc localhost 5000.
I always get an EOF error when reading the data, however. I checked the docs and this is only supposed to happen when there is no more input available, however I don't understand why this is happening here.
This is my code:
package main
import (
"bufio"
"fmt"
"net"
"os"
)
// Connection details
type connection struct {
host string
port string
network string
}
// Initialise a Listener on a given port
// Pass handling into seperate goroutine
func main() {
localConn := connection{
host: "", // Localhost
port: "5000",
network: "tcp",
}
listener, err := net.Listen(localConn.network, localConn.host+":"+localConn.port)
checkError("Error listening: ", err)
conn, err := listener.Accept()
for {
checkError("Error accepting: ", err)
go handleRequest(conn)
}
}
// Delegate handling of requests
func handleRequest(conn net.Conn) {
// Read message up until newline delimiter
message, err := bufio.NewReader(conn).ReadString('\n')
checkError("Error reading: ", err)
fmt.Println("Message recieved: ", string(message))
conn.Write([]byte("Recieved message: " + string(message) + "\n"))
conn.Close()
}
// Check if an error exists
// If so, print and exit program. (Not super robust!)
func checkError(message string, err error) {
if err != nil {
fmt.Println(message, err.Error())
os.Exit(1)
}
}
You appear to be using echo incorrectly. The flag -e interprets the two characters \n as a newline (check here).
Use the following command to send data to server:
echo -e "Server test.\n" | nc localhost 5000
Other than that you should also fix for loop:
for {
conn, err := listener.Accept()
checkError("Error accepting: ", err)
go handleRequest(conn)
}
In your original code you are only one connection is ever accepted. After that the for loop simply starts more goroutines that try to read on a closed connection (error or not, first handleRequest call closes the connection).
One problem is on these lines:
conn, err := listener.Accept()
for {
checkError("Error accepting: ", err)
go handleRequest(conn)
}
The application starts goroutines in a loop to read the single connection. The first goroutine to read the connection is successful. The subsequent goroutines report an error.
Change the code to:
for {
conn, err := listener.Accept()
checkError("Error accepting: ", err)
go handleRequest(conn)
}
The client is not sending a newline as expected by the server. Use this command to send the message:
echo "Server test." | nc localhost 5000