Can I increase Golang's http stream chunk size? - http

Question

I'm trying to send data (files or whatever) through HTTP from the client to a server and read them as stream in the server.
But I noticed the chunk size or buffer size when the request's body is read it is fixed to 32kb. I tried doing it with TCP before using HTTP and the buffer size was the expected assigned size.
The data received from the request is being written to a file
Questions:
Is it possible to increase the chunk / buffer size?
if it is possible, by having a bigger buffer size will it increase performance due to less write calls to to the file being created?
If it is not possible, should I worry about performance loss by doing more write calls to the file being created?
Would it be better to use TCP? I really need the headers and http response
Here is some code for illustration:
client.go:
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
)
func main() {
addr := "http://localhost:8080"
path := "path/to/file"
sendHTTP(addr, path)
}
func sendHTTP(addr, path string) {
f, err := os.Open(path)
if err != nil {
log.Fatal("Error opening file:", err)
}
client := &http.Client{}
req, err := http.NewRequest("POST", addr, f)
if err != nil {
f.Close()
log.Fatal("Error creating request:", err)
}
_, err = client.Do(req)
if err != nil {
f.Close()
log.Fatal("Error doing request:", err)
}
}
server.go:
package main
import (
"fmt"
"io"
"log"
"net/http"
)
func main() {
addr := ":8080"
http.HandleFunc("/", handler)
http.ListenAndServe(addr, nil)
}
func handler(_ http.ResponseWriter, r *http.Request) {
buf := make([]byte, 512*1024) // 512kb
for {
br, err := r.Body.Read(buf)
if err == io.EOF {
break
} else if err != nil {
log.Println("Error reading request:", err)
break
}
fmt.Println(br) // is always 32kb
}
}

Answer 1

The call r.Body.Read(buf) waits for data from the network and returns up to len(buf) bytes of the available data. The amount of available data at the time of the call depends on timing and buffer sizes on the client, server and network. It's not easy to control.
The data received from the request is being written to a file
To write the data to the file in the most efficient way, copy from the request body to the file using io.Copy. Here's an example where f is the *os.File you want to write:
_, err := io.Copy(f, r.Body)
if err != nil {
// handle error
}
At the time I am writing this answer, the io.Copy function calls f.ReadFrom(r.Body) to copy the request body to a file.