This question already has answers here:
Limit bytes to read from HTTP response
(3 answers)
Limiting amount of data read in the response to a HTTP GET request
(2 answers)
Closed 9 months ago.
I want to discard all bytes over a given limit during reading of HTTP Response (skip too large response) using LimitedReader.
Is this the correct way to do that?
func getRequestData(req *http.Request, client *http.Client,
responseSizeLimit int64) ([]byte, error) {
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer func() {
io.Copy(ioutil.Discard, resp.Body) // response body must be read to the end and closed
resp.Body.Close()
}()
buffer := bufio.NewReader(resp.Body)
reader := io.LimitReader(buffer, responseSizeLimit)
return ioutil.ReadAll(reader)
}
And what if I want to return io.Reader instead of []byte, is it possible for http.Response.Body and limitedReader ?
Related
So my HTTP client initialisation and send request code looks like this.
package http_util
import (
"crypto/tls"
"net/http"
"time"
)
var httpClient *http.Client
func Init() {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
MaxIdleConnsPerHost: 200,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
}
httpClient = &http.Client{Transport: tr, Timeout: 30 * time.Second}
}
func SendRequest(ctx context.Context, request *http.Request) (*SomeRespStruct, error) {
httpResponse, err := httpClient.Do(request)
if err != nil {
return nil, err
}
responseBody, err := ioutil.ReadAll(httpResponse.Body)
defer httpResponse.Body.Close()
if err != nil {
return nil, err
}
response := &SomeRespStruct{}
err = json.Unmarshal(responseBody, response)
if err != nil {
return nil, err
}
return response, nil
}
When I launch my server, I call http_util.Init().
The issue arises when I receive multiple requests (20+) at once to call this external server. In one of my functions I do
package external_api
import (
"context"
"log"
)
func SomeAPICall(ctx context.Context) (SomeRespStruct, error) {
// Build request
request := buildHTTPRequest(...)
log.Printf("Send request: %v", request)
response, err := http_util.SendRequest(ctx, request)
// Error checks
if err != nil {
log.Printf("HTTP request timed out: %v", err)
return nil, err
}
log.Printf("Received response: %v", response)
return response, nil
}
My issue is that I get a 15~20s lag in between the Send request and Received response logs based on the output timestamp when there is high request volume. Upon checking with the server that's handling my requests, I found out that on their end, processing time from end-to-end takes less than a second (the same exact request that had a long turnaround time according to my own logs), so I'm not too sure what is the root cause of this high turnaround time. I also did a traceroute and a ping to the server as well and there was no delay, so this should not be a network error.
I've looked around and it seems like the suggested solutions are:
to increase the MaxIdleConnsPerHost
to read the HTTP response body in full and close it
Both of which I have already done.
I'm not sure if there is more tuning to be done regarding the configuration of my HTTP client to resolve this issue, or if I should investigate other workarounds, for instance retry or perhaps scaling (but my CPU and memory utilisation are at the 2-3% range).
I'm perpetually caught out by reading an io.ReadCloser and then forgetting that I've read it before, and when I read it again, I get an empty payload. I wish there was some lint check for my stupidity. Nonetheless, I think I can use TeeReader, but it fails to meet my expectations here:
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
buf := &bytes.Buffer{}
tee := io.TeeReader(r.Body, buf)
body, err := ioutil.ReadAll(tee)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
log.Println("body", string(body))
payload, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("dump request", string(payload))
w.WriteHeader(http.StatusOK)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
The body is missing from my "dump request" log line.
I.e. when I run curl -i -X POST --data '{"username":"xyz","password":"xyz"}' http://localhost:8080
I want the original request in full:
2019/01/14 11:09:50 dump request POST / HTTP/1.1
Host: localhost:8080
Accept: */*
Content-Length: 35
Content-Type: application/x-www-form-urlencoded
User-Agent: curl/7.63.0
{"username":"xyz","password":"xyz"}
What am I missing?
You cannot rewind an io.ReadCloser, unless the underlying value is also an io.ReadSeeker.
An io.ReadCloser, by very definition, has exactly two methods: Read and Close. So there is obviously there is no option to rewind.
An io.ReadSeeker, by contrast, has two methods: Read and Seek, the latter which allows rewinding.
If you need to accept only io.ReadClosers which are also seekable, you can easily combine these two:
type ReadSeekCloser interface {
io.Reader
io.Seeker
io.Closer
}
Now you can use your custom ReadSeekCloser type in place of an io.ReadCloser, and you'll have the option to rewind your reader.
Of course, few io.ReadClosers in the wild actually conform to this interface (os.File will be the main one that does). If you have an io.ReadCloser that does not implement the Seek method (such as a network stream), probably the easiest way to make it Seekable is to dump the contents to a file, then open that file. There would be other ways to make an in-memory buffer seekable (bytes.NewReader for instance), but variations will require reading the stream into memory or onto disk, first.
How do I rewind a io.ReadCloser in Go[...]?
You cannot. A ReadCloser can be read and closed. Unless the actual underlying type has some method to rewind you simply cannot.
(For your case you may just use the bytes.Buffer, possibly after adding a Close method via io/ioutil.ReadCloser as the Request.Body; but this is not "rewinding" but "replacing".)
https://golang.org/pkg/net/http/httputil/#DumpRequest
DumpRequest returns the given request in its HTTP/1.x wire representation. It should only be used by servers to debug client requests.
Clearly DumpRequest is for Dubug use.
But if you don't care about that. The godoc also metioned:
If body is true, DumpRequest also returns the body. To do so, it consumes req.Body and then replaces it with a new io.ReadCloser that yields the same bytes.
So you can call DumpRequest first then ReadAll from Body, cause the Body is still the same after DumpRequest.
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
payload, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("dump request", string(payload))
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
log.Println("body", string(body))
w.WriteHeader(http.StatusOK)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
Request bodies aren't seek-able. There is no rewind, they aren't buffered in memory. When that data is read, it is read from the network stream. Imagine a large upload, buffering all of that data in memory by default would be wasteful.
That said, there are some things you can do to get the output you desire. It involves replacing r.Body after you've read it.
You mention wishing for a lint check. I've found that declaring a buffer and using only that buffer can help. If you like, you can even replace r.Body with that buffer. You will still need to remember to "rewind" with Seek.
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
b := &bytes.Buffer{}
_, err := io.Copy(b, r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
bodyCopy := bytes.NewReader(b.Bytes())
log.Println("body", b.String())
r.Body = io.NopCloser(bodyCopy)
bodyCopy.Seek(0, 0)
payload, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("dump request", string(payload))
w.WriteHeader(http.StatusOK)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
When throwing an HTTP Request with Go and receiving a Response, I want to receive a response while streaming, considering the case where the ResponseBody is huge (1 GB or more).
resp, err: = http.Client.Do(req)
In this case, if the body is huge, I can not read the Header and I do not know the state of Response.
Is there any solution?
(Edit: If you're unable to get the "Content-length" header from the response, it is possible that the web service you're hitting doesn't return that header. In such a case, there's no way to know the length of the response body without reading it completely. You can simulate that in the following example by removing the line that sets the Content-length header in the response.)
The standard Go net/http package handles large responses very well. Here's a self contained example to demonstrate:
// Start a mock HTTP server that returns 2GB of data in the response. Make a
// HTTP request to this server and print the amount of data read from the
// response.
package main
import (
"fmt"
"io"
"log"
"net/http"
"strings"
"time"
)
const oneMB = 1024 * 1024
const oneGB = 1024 * oneMB
const responseSize = 2 * oneGB
const serverAddr = "localhost:9999"
func startServer() {
// Mock HTTP server that always returns 2GB of data
go http.ListenAndServe(serverAddr, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-length", fmt.Sprintf("%d", responseSize))
// 1MB buffer that'll be copied multiple times to the response
buf := []byte(strings.Repeat("x", oneMB))
for i := 0; i < responseSize/len(buf); i++ {
if _, err := w.Write(buf); err != nil {
log.Fatal("Failed to write to response. Error: ", err.Error())
}
}
}))
// Some grace period for the server to start
time.Sleep(100 * time.Millisecond)
}
func main() {
startServer()
// HTTP client
req, err := http.NewRequest("GET", "http://"+serverAddr, nil)
if err != nil {
log.Fatal("Error creating HTTP request: ", err.Error())
}
client := http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal("Error making HTTP request: ", err.Error())
}
// Read the response header
fmt.Println("Response: Content-length:", resp.Header.Get("Content-length"))
bytesRead := 0
buf := make([]byte, oneMB)
// Read the response body
for {
n, err := resp.Body.Read(buf)
bytesRead += n
if err == io.EOF {
break
}
if err != nil {
log.Fatal("Error reading HTTP response: ", err.Error())
}
}
fmt.Println("Response: Read", bytesRead, "bytes")
}
You wouldn't want to read the entire response in memory if it's too large. Write it to a temporary file instead and then process that.
If instead you're looking for options to do this reliably when the network isn't very reliable, look for "HTTP range requests" using which you can resume partially downloaded data.
This question already has answers here:
golang http+jsonrpc access from web page
(2 answers)
Closed 5 years ago.
How can I use JSON-RPC over HTTP based on this specification in Go?
Go provides JSON-RPC codec in net/rpc/jsonrpc but this codec use network connection as input so you cannot use it with go RPC HTTP handler. I attach sample code that uses TCP for JSON-RPC:
func main() {
cal := new(Calculator)
server := rpc.NewServer()
server.Register(cal)
listener, e := net.Listen("tcp", ":1234")
if e != nil {
log.Fatal("listen error:", e)
}
for {
if conn, err := listener.Accept(); err != nil {
log.Fatal("accept error: " + err.Error())
} else {
log.Printf("new connection established\n")
go server.ServeCodec(jsonrpc.NewServerCodec(conn))
}
}
}
The built-in RPC HTTP handler uses the gob codec on a hijacked HTTP connection. Here's how to do the same with the JSONRPC.
Write an HTTP handler that runs the JSONRPC server with a hijacked connection.
func serveJSONRPC(w http.ResponseWriter, req *http.Request) {
if req.Method != "CONNECT" {
http.Error(w, "method must be connect", 405)
return
}
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
http.Error(w, "internal server error", 500)
return
}
defer conn.Close()
io.WriteString(conn, "HTTP/1.0 Connected\r\n\r\n")
jsonrpc.ServeConn(conn)
}
Register this handler with the HTTP server. For example:
http.HandleFunc("/rpcendpoint", serveJSONRPC)
EDIT: OP has since updated question to make it clear that they want GET/POST instead of connect.
I'm trying to send data (files or whatever) through HTTP from the client to a server and read them as stream in the server.
But I noticed the chunk size or buffer size when the request's body is read it is fixed to 32kb. I tried doing it with TCP before using HTTP and the buffer size was the expected assigned size.
The data received from the request is being written to a file
Questions:
Is it possible to increase the chunk / buffer size?
if it is possible, by having a bigger buffer size will it increase performance due to less write calls to to the file being created?
If it is not possible, should I worry about performance loss by doing more write calls to the file being created?
Would it be better to use TCP? I really need the headers and http response
Here is some code for illustration:
client.go:
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
)
func main() {
addr := "http://localhost:8080"
path := "path/to/file"
sendHTTP(addr, path)
}
func sendHTTP(addr, path string) {
f, err := os.Open(path)
if err != nil {
log.Fatal("Error opening file:", err)
}
client := &http.Client{}
req, err := http.NewRequest("POST", addr, f)
if err != nil {
f.Close()
log.Fatal("Error creating request:", err)
}
_, err = client.Do(req)
if err != nil {
f.Close()
log.Fatal("Error doing request:", err)
}
}
server.go:
package main
import (
"fmt"
"io"
"log"
"net/http"
)
func main() {
addr := ":8080"
http.HandleFunc("/", handler)
http.ListenAndServe(addr, nil)
}
func handler(_ http.ResponseWriter, r *http.Request) {
buf := make([]byte, 512*1024) // 512kb
for {
br, err := r.Body.Read(buf)
if err == io.EOF {
break
} else if err != nil {
log.Println("Error reading request:", err)
break
}
fmt.Println(br) // is always 32kb
}
}
The call r.Body.Read(buf) waits for data from the network and returns up to len(buf) bytes of the available data. The amount of available data at the time of the call depends on timing and buffer sizes on the client, server and network. It's not easy to control.
The data received from the request is being written to a file
To write the data to the file in the most efficient way, copy from the request body to the file using io.Copy. Here's an example where f is the *os.File you want to write:
_, err := io.Copy(f, r.Body)
if err != nil {
// handle error
}
At the time I am writing this answer, the io.Copy function calls f.ReadFrom(r.Body) to copy the request body to a file.