Go's http.MaxBytesReader, why pass in writer? - http

Intuitively, I would think that when you create a MaxByteReader and pass in the http.ResponseWriter, it would write out the status code for you. But that isn't the case, what does the writer actually do?
example:
func maxBytesMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, 1)
next.ServeHTTP(w, r)
})
}
func mainHandler(w http.ResponseWriter, r *http.Request) {
var i interface{}
err := json.NewDecoder(r.Body).Decode(&i)
if err != nil {
fmt.Println(err.Error())
}
}
func TestMaxBytesMiddleware(t *testing.T) {
handlerToTest := maxBytesMiddleware(http.HandlerFunc(mainHandler))
req := httptest.NewRequest(http.MethodPost, "http://test.com", bytes.NewReader(json.RawMessage(`{"hello":"world"}`)))
recorder := httptest.NewRecorder()
handlerToTest.ServeHTTP(recorder, req)
if recorder.Result().StatusCode != http.StatusRequestEntityTooLarge {
t.Errorf("expected %d got %d", http.StatusRequestEntityTooLarge, recorder.Result().StatusCode)
}
}
but when this test runs I get this:
http: request body too large
--- FAIL: TestMaxBytesMiddleware (0.00s)
main_test.go:37: expected 413 got 200
if I want the desired functionality of what I thought this function did, I need to change my mainHandler to something like this:
func mainHandler(w http.ResponseWriter, r *http.Request) {
var i interface{}
err := json.NewDecoder(r.Body).Decode(&i)
if err != nil {
if err.Error() == "http: request body too large" {
w.WriteHeader(http.StatusRequestEntityTooLarge)
return
}
fmt.Println(err.Error())
}
}
So what is that writer even there for?

If the MaxBytesReader stops before reading the whole body, it sets some flags on the writer that make sure that the HTTP connection will be closed after the response is sent. Normally the server would be willing to read another request from the same connection (HTTP keepalive), but it can't do that if there are unread bits of the previous request still in the pipeline, so it has to close the connection, forcing the client to make a new connection if it wants to send more requests.
This is accomplished using the private requestTooLarge method of http.ResponseWriter.

Related

Write pipe reading into http response in golang

Here is the schema :
Client sends a POST request to server A
server A process this and sends a GET to server B
server B sends a response through A to the client
I though the best idea was to make a pipe which would read the response of the GET, and write into the response of the POST, but I got many types problems.
func main() {
r := mux.NewRouter()
r.HandleFunc("/test/{hash}", testHandler)
log.Fatal(http.ListenAndServe(":9095", r))
}
func handleErr(err error) {
if err != nil {
log.Fatalf("%s\n", err)
}
}
func testHandler(w http.ResponseWriter, r *http.Request){
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
write, _ = http.Get("http://localhost:9090/test/" + hash)
defer write.Close()
}()
w.Write(read)
}
When I run this I get the following error:
./ReverseProxy.go:61: cannot use read (type *io.PipeReader) as type []byte in argument to w.Write
Is there a way, to properly insert a io.PipeReader format into an http response?
Or am I doing this in a totally wrong way?
You are not actually writing to it, you're replacing the pipe's write.
Something along the lines of:
func testHandler(w http.ResponseWriter, r *http.Request) {
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
defer write.Close()
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
return
}
defer resp.Body.Close()
io.Copy(write, resp.Body)
}()
io.Copy(w, read)
}
Although, I agree with #JimB, for this instance, the pipe isn't even needed, something like this should be more efficient:
func testHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
hash := vars["hash"]
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
// handle error
return
}
defer resp.Body.Close()
io.Copy(w, resp.Body)
}

Terminating or aborting an HTTP request

What's the way to abort my API serving with some error message?
Link to call my service:
http://creative.test.spoti.io/api/getVastPlayer?add=
{"Json":Json}&host=api0.spoti.io&domain=domain&userAgent=userAgent&mobile=true
To call my service the client need to send a Json and some params.
I want to test if the params that I get are correct, if not I want send a error message.
The response should be a Json Code {"Result":"Result","Error":"error message"}
I tried log.fatal and os.Exit(1) they stop the service, not just the call request. panic aborts the call but it prevents me to send a http.ResponseWriter which is my error message.
I read something about panic, defer, recover but I don't really know how can I use them to solve this problem.
return works:
mobile :=query.Get("mobile")
if mobile=="mobile" {
str:=`{"Resultt":"","Error":"No valide Var"}`
fmt.Fprint(w, str)
fmt.Println("No successfull Operation!!")
return}
But I can use it just in the main function, because in the other functions it exits just the func not the caller function (request).
Terminating the serving of an HTTP request is nothing more than to return from the ServeHTTP() method, e.g.:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// examine incoming params
if !ok {
str := `{"Result":"","Error":"No valide Var"}`
fmt.Fprint(w, str)
return
}
// Do normal API serving
})
panic(http.ListenAndServe(":8080", nil))
Notes:
If the input params of your API service are invalid, you should consider returning an HTTP error code instead of the implied default 200 OK. For this you can use the http.Error() function, for example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// examine incoming params
if !ok {
http.Error(w, `Invalid input params!`, http.StatusBadRequest)
return
}
// Do normal API serving
})
For a more sophisticated example where you send back JSON data along with the error code:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// examine incoming params
if !ok {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadRequest)
str := `{"Result":"","Error":"No valide Var"}`
fmt.Fprint(w, str)
return
}
// Do normal API serving
})
Example showing how to propagate "returning"
If the error is detected outside of ServeHTTP(), e.g. in a function that is called from ServeHTTP(), you have to return this error state so that ServeHTTP() can return.
Let's assume you have the following custom type for your required parameters and a function which is responsible to decode them from a request:
type params struct {
// fields for your params
}
func decodeParams(r *http.Request) (*params, error) {
p := new(params)
// decode params, if they are invalid, return an error:
if !ok {
return nil, errors.New("Invalid params")
}
// If everything goes well:
return p, nil
}
Using these:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
p, err := decodeParams(r)
if err != nil {
http.Error(w, `Invalid input params!`, http.StatusBadRequest)
return
}
// Do normal API serving
})
Also see this related question: Golang, how to return in func FROM another func?

In go, how to inspect the http response that is written to http.ResponseWriter?

There's probably something obvious that I'm missing but I'm trying to debug the HTTP response written by my go server.
I see that there's httputil.DumpResponse available but it takes a http.Response object and what I have available is http.ResponseWriter
Is there a way to extract the http.Response from http.ResponseWriter so I can inspect the content of the response to console or log?
Context:
I'm writing a simple server-side authentication using https://github.com/RangelReale/osin and it's default example, but could not understand why the front-end (using http://ember-simple-auth.com) interprets a failed authentication (incorrect password) as success.
Here's the snippet:
r = mux.NewRouter()
r.HandleFunc("/token", func (w http.ResponseWriter, r *http.Request) {
fmt.Printf("r.HandleFunc /token\n")
resp := server.NewResponse()
defer resp.Close()
r.ParseForm()
grantType := r.FormValue("grant_type")
username := r.FormValue("username")
password := r.FormValue("password")
fmt.Printf("/token : grantType=%s username=%s password=%s\n", grantType, username, password)
if ar := server.HandleAccessRequest(resp, r); ar != nil {
if username == "user" && password == "correct-password" {
ar.Authorized = true
} else {
ar.Authorized = false
}
server.FinishAccessRequest(resp, r, ar)
}
osin.OutputJSON(resp, w, r)
// Debug - doesn't work yet
dump, err := httputil.DumpResponse(w, true)
if err != nil {
fmt.Printf("%s\n", dump)
}
});
http.Handle("/token", r)
Write to an *httptest.ResponseRecorder (which implements http.ResponseWriter) and inspect it.
Example from the package:
package main
import (
"fmt"
"log"
"net/http"
"net/http/httptest"
)
func main() {
handler := func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "something failed", http.StatusInternalServerError)
}
req, err := http.NewRequest("GET", "http://example.com/foo", nil)
if err != nil {
log.Fatal(err)
}
w := httptest.NewRecorder()
handler(w, req)
fmt.Printf("%d - %s", w.Code, w.Body.String())
}
Edit to answer question in comments:
If I understand your question correctly, then yes, you can make use of closures for this.
Consider the following to be your handler:
func MyHandler(w http.ResponseWriter, r *http.Request) {
// do useful stuff...
}
You could then register the following closure with your servemux to attain the desired effect:
http.HandleFunc("/my/url", func(w http.ResponseWriter, r *http.Request) {
// first call MyHandler
MyHandler(w, r)
// then log whatever you need
log.Printf("%#v\n", w)
})
If this pattern proves useful to you then you could write a higher-order method that wraps any func(http.ResponseWriter, *http.Request) in such a closure. That's a topic for itself, though.

about http hijacking and keep-alive

i use
resp, err := http.Get("http://example.com/")
get a http.Response, and i want to exactly write to a http handler, but only http.ResponseWriter, so i hijack it.
...
webConn, webBuf, err := hj.Hijack()
if err != nil {
// handle error
}
defer webConn.Close()
// Write resp
resp.Write(webBuf)
...
Write raw request
But When i hijack, http connection can't reuse (keep-alive), so it slow.
How to solve?
Thanks! Sorry for my pool English.
update 12/9
keep-alive, It keep two tcp connection, and can reuse.
but when i hijack, and conn.Close(), It can't reuse old connection, so it create a new tcp connection when i each refresh.
Do not use hijack, Because once hijack, the HTTP server library will not do anything else with the connection, So can't reuse.
I change way, copy Header and Body, look like reverse proxy (http://golang.org/src/pkg/net/http/httputil/reverseproxy.go), Is works.
Example:
func copyHeader(dst, src http.Header) {
for k, w := range src {
for _, v := range w {
dst.Add(k, v)
}
}
}
func copyResponse(r *http.Response, w http.ResponseWriter) {
copyHeader(w.Header(), r.Header)
w.WriteHeader(r.StatusCode)
io.Copy(w, r.Body)
}
func handler(w http.ResponseWriter, r *http.Response) {
resp, err := http.Get("http://www.example.com")
if err != nil {
// handle error
}
copyResponse(resp, w)
}
It seem that once the connection is closed the keep-alive connection closes as well.
One possible solution would be to prevent the connection from closing until desired, but I'm not sure if that good advise.
Maybe the correct solution involves creating a instance of net.TCPConn, copying the connection over it, then calling .SetKeepAlive(true).
Before running the below example, launch another terminal with netstat -antc | grep 9090.
Routes in example:
localhost:9090/ok is a basic (non-hijacked) connection
localhost:9090 is a hijacked connection, lasting for 10 seconds.
Example
package main
import (
"fmt"
"net/http"
"sync"
"time"
)
func checkError(e error) {
if e != nil {
panic(e)
}
}
var ka_seconds = 10
var conn_id = 0
func main() {
http.HandleFunc("/ok", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "ok")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
conn_id++
fmt.Printf("Connection %v: Keep-alive is enabled %v seconds\n", conn_id, ka_seconds)
hj, ok := w.(http.Hijacker)
if !ok {
http.Error(w, "webserver doesn't support hijacking", http.StatusInternalServerError)
return
}
conn, bufrw, err := hj.Hijack()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Don't forget to close the connection:
time.AfterFunc(time.Second* time.Duration(ka_seconds), func() {
conn.Close()
fmt.Printf("Connection %v: Keep-alive is disabled.\n", conn_id)
})
resp, err := http.Get("http://www.example.com")
checkError(err)
resp.Write(bufrw)
bufrw.Flush()
})
fmt.Println("Listing to localhost:9090")
http.ListenAndServe(":9090", nil)
}
Related issue: http://code.google.com/p/go/issues/detail?id=5645

Not buffered http.ResponseWritter in Golang

I'm writing a simple web app in Go and I want my responses to be streamed to the client (i.e. not buffered and sent in blocks once the request is fully processed) :
func handle(res http.ResponseWriter, req *http.Request) {
fmt.Fprintf(res, "sending first line of data")
sleep(10) //not real code
fmt.Fprintf(res, "sending second line of data")
}
From the client point of view, the two lines will be sent at the same time. Any suggestions are appreciated :)
Edit after #dystroy answer
It's possible to flush after each write I personally make, but in my use case it's not enough:
cmd := exec.Command("a long command that outputs lots of lines")
cmd.Stdout = res //where res is a http.ResponseWritter
cmd.Stderr = res
err := cmd.Run()
I want the output of my cmd to be flushed as well. Anyway to "autoflush" the ResponseWritter ?
Solution
I found help on golang's mailing list. There is 2 way to achieve this: using hijacker that allow to take over the underlying TCP connection of HTTP, or piping the stdout and stderr of the command in a go routine that will write and flush :
pipeReader, pipeWriter := io.Pipe()
cmd.Stdout = pipeWriter
cmd.Stderr = pipeWriter
go writeCmdOutput(res, pipeReader)
err := cmd.Run()
pipeWriter.Close()
//---------------------
func writeCmdOutput(res http.ResponseWriter, pipeReader *io.PipeReader) {
buffer := make([]byte, BUF_LEN)
for {
n, err := pipeReader.Read(buffer)
if err != nil {
pipeReader.Close()
break
}
data := buffer[0:n]
res.Write(data)
if f, ok := res.(http.Flusher); ok {
f.Flush()
}
//reset buffer
for i := 0; i < n; i++ {
buffer[i] = 0
}
}
}
Last update
Even nicer: http://play.golang.org/p/PpbPyXbtEs
As implied in the documentation, some ResponseWriter may implement the Flusher interface.
This means you can do something like this :
func handle(res http.ResponseWriter, req *http.Request) {
fmt.Fprintf(res, "sending first line of data")
if f, ok := res.(http.Flusher); ok {
f.Flush()
} else {
log.Println("Damn, no flush");
}
sleep(10) //not real code
fmt.Fprintf(res, "sending second line of data")
}
Be careful that buffering can occur in many other places in the network or client side.
Sorry if I've misunderstood your question, but would something like the below do the trick?
package main
import (
"bytes"
"fmt"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
body := make([]byte, int(r.ContentLength))
b := bytes.NewBuffer(body)
if _, err := b.ReadFrom(r.Body); err != nil {
fmt.Fprintf(w, "%s", err)
}
if _, err := b.WriteTo(w); err != nil {
fmt.Fprintf(w, "%s", err)
}
}
func main() {
http.HandleFunc("/", handler)
if err := http.ListenAndServe(":8080", nil); err != nil {
panic(err)
}
}
$ curl --data "param1=value1&param2=value2" http://localhost:8080
returns:
param1=value1&param2=value2
You could always append whatever data you wanted to body, or read more bytes into the buffer from elsewhere before writing it all out again.

Resources