http.ResponseWriter.WriteHeader() caused deadlock - http

I was trying to use the httptest package in golang. I found out something I don't understand. Here is the code:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
)
func main() {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello1"))
}))
ts.Close()
ts = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello2"))
}))
ts.Close()
ts = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(100)
w.Write([]byte("Hello3"))
}))
res, err := http.Get(ts.URL)
if err != nil {
log.Fatal(err)
}
greeting, err := ioutil.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
ts.Close()
fmt.Printf("%s", greeting)
}
In this code example, I was trying to open and close httptest servers several times. Somehow it caused deadlock in The Go Playground. I tried on my own environment (Go version: go1.7.4 darwin/amd64) and it caused hanging without responding at all.
My question is: Why w.WriteHeader(100) caused deadlock but w.WriteHeader(200) doesn't? Is it the bug from the core library of Golang or just I misunderstood some usage? Tks!

If you slightly modify code:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
)
func main() {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello1"))
}))
ts.Close()
ts = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello2"))
}))
ts.Close()
ts = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(100)
w.Write([]byte("Hello3"))
}))
fmt.Println("before get") ///// . <----
res, err := http.Get(ts.URL)
fmt.Println("after get") ///// . <----
if err != nil {
log.Fatal(err)
}
greeting, err := ioutil.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
ts.Close()
fmt.Printf("%s", greeting)
}
and run it - you'll see only first line.
That means Go http client want more data from you. So it hangs on line
res, err := http.Get(ts.URL)
and cannot get to the ts.Close() below.
Next - lets modify a test, so it will close a connection and this way release a clients waiting lock:
ts = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(100)
w.Write([]byte("Hello3"))
hj, _ := w.(http.Hijacker)
conn, _, _ := hj.Hijack()
conn.Close()
}))
I close connection explicitly but this way you get both check strings and test finishes ok. Try it.
Of course it's a HTTP protocol violation so I get an error:
Get http://127.0.0.1:54243: net/http: HTTP/1.x transport connection broken: unexpected EOF

Related

How to Write Files or Folder to IPFS via the HTTP API

I'm confused about the HTTP API docs of IPFS。next is part of it。
/api/v0/add
Add a file or directory to IPFS.
//but how to add a directory by golang? it look like so simple but no a example to finish it
#cURL Example
curl -X POST -F file=#myfile "http://127.0.0.1:5001/api/v0/add?quiet=&quieter=&silent=&progress=&trickle=&only-hash=&wrap-with-directory=&chunker=size-262144&pin=true&raw-leaves=&nocopy=&fscache=&cid-version=&hash=sha2-256&inline=&inline-limit=32"
I worked on the same issue and found this working shell solution:
https://community.infura.io/t/ipfs-http-api-add-directory/189/8
you can rebuild this in go
package main
import (
"bytes"
"github.com/stretchr/testify/assert"
"io"
"io/ioutil"
"mime/multipart"
"net/http"
"os"
"strings"
"testing"
)
func TestUploadFolderRaw(t *testing.T) {
ct, r, err := createForm(map[string]string{
"/file1": "#/my/path/file1",
"/dir": "#/my/path/dir",
"/dir/file": "#/my/path/dir/file",
})
assert.NoError(t, err)
resp, err := http.Post("http://localhost:5001/api/v0/add?pin=true&recursive=true&wrap-with-directory=true", ct, r)
assert.NoError(t, err)
respAsBytes, err := ioutil.ReadAll(resp.Body)
assert.NoError(t, err)
t.Log(string(respAsBytes))
}
func createForm(form map[string]string) (string, io.Reader, error) {
body := new(bytes.Buffer)
mp := multipart.NewWriter(body)
defer mp.Close()
for key, val := range form {
if strings.HasPrefix(val, "#") {
val = val[1:]
file, err := os.Open(val)
if err != nil { return "", nil, err }
defer file.Close()
part, err := mp.CreateFormFile(key, val)
if err != nil { return "", nil, err }
io.Copy(part, file)
} else {
mp.WriteField(key, val)
}
}
return mp.FormDataContentType(), body, nil
}
or use https://github.com/ipfs/go-ipfs-http-client which seems to be a better way. I'm working on it and tell you when I know how to use it
Greetings

How I can return multi response

I've read about asynchronous response and,
I am trying to return multi response using golang,
but I don't know how I can do that.
my example:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
http.HandleFunc("/", multiResponse)
http.ListenAndServe(":8080", nil)
}
func multiResponse(w http.ResponseWriter, r *http.Request) {
go func() {
resp, err := http.Get("https://www.google.com.eg/search?q=hello")
if err != nil {
log.Println(err)
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Println(err)
}
// w.WriteHeader(http.StatusOK)
// w.Write(body)
fmt.Fprintln(w, string(body))
resp.Body.Close()
fmt.Println("done...")
}()
w.WriteHeader(http.StatusAccepted)
}
is there any examples to understand asynchronous response?
Edit:
I meaning about multiple response,
when I receive request from client I will send StatusAccepted immediately,
and in another goroutine other process work, then after finishing send StatusOK with some data.

In go, how to inspect the http response that is written to http.ResponseWriter?

There's probably something obvious that I'm missing but I'm trying to debug the HTTP response written by my go server.
I see that there's httputil.DumpResponse available but it takes a http.Response object and what I have available is http.ResponseWriter
Is there a way to extract the http.Response from http.ResponseWriter so I can inspect the content of the response to console or log?
Context:
I'm writing a simple server-side authentication using https://github.com/RangelReale/osin and it's default example, but could not understand why the front-end (using http://ember-simple-auth.com) interprets a failed authentication (incorrect password) as success.
Here's the snippet:
r = mux.NewRouter()
r.HandleFunc("/token", func (w http.ResponseWriter, r *http.Request) {
fmt.Printf("r.HandleFunc /token\n")
resp := server.NewResponse()
defer resp.Close()
r.ParseForm()
grantType := r.FormValue("grant_type")
username := r.FormValue("username")
password := r.FormValue("password")
fmt.Printf("/token : grantType=%s username=%s password=%s\n", grantType, username, password)
if ar := server.HandleAccessRequest(resp, r); ar != nil {
if username == "user" && password == "correct-password" {
ar.Authorized = true
} else {
ar.Authorized = false
}
server.FinishAccessRequest(resp, r, ar)
}
osin.OutputJSON(resp, w, r)
// Debug - doesn't work yet
dump, err := httputil.DumpResponse(w, true)
if err != nil {
fmt.Printf("%s\n", dump)
}
});
http.Handle("/token", r)
Write to an *httptest.ResponseRecorder (which implements http.ResponseWriter) and inspect it.
Example from the package:
package main
import (
"fmt"
"log"
"net/http"
"net/http/httptest"
)
func main() {
handler := func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "something failed", http.StatusInternalServerError)
}
req, err := http.NewRequest("GET", "http://example.com/foo", nil)
if err != nil {
log.Fatal(err)
}
w := httptest.NewRecorder()
handler(w, req)
fmt.Printf("%d - %s", w.Code, w.Body.String())
}
Edit to answer question in comments:
If I understand your question correctly, then yes, you can make use of closures for this.
Consider the following to be your handler:
func MyHandler(w http.ResponseWriter, r *http.Request) {
// do useful stuff...
}
You could then register the following closure with your servemux to attain the desired effect:
http.HandleFunc("/my/url", func(w http.ResponseWriter, r *http.Request) {
// first call MyHandler
MyHandler(w, r)
// then log whatever you need
log.Printf("%#v\n", w)
})
If this pattern proves useful to you then you could write a higher-order method that wraps any func(http.ResponseWriter, *http.Request) in such a closure. That's a topic for itself, though.

Go : add logging to each router

Go : add logging to each router
I want to log all my network request in Go web app.
Something like negroni:
// https://github.com/codegangsta/negroni/blob/master/logger.go
// NewLogger returns a new Logger instance
func NewLogger() *Logger {
return &Logger{log.New(os.Stdout, "[negroni] ", 0)}
}
func (l *Logger) ServeHTTP(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc) {
start := time.Now()
l.Printf("Started %s %s", r.Method, r.URL.Path)
next(rw, r)
res := rw.(ResponseWriter)
l.Printf("Completed %v %s in %v", res.Status(), http.StatusText(res.Status()), time.Since(start))
}
So here's my code:
router := httprouter.New()
handler := func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
type Page struct {
Title string
}
tp := template.Must(template.ParseFiles("templates/main.html", "templates/base.html"))
err := tp.ExecuteTemplate(w, "base", &Page{Title: "AAA"})
if err != nil {
log.Fatal(err)
}
router.Handle("GET", "/", handler)
l := log.New(os.Stdout, "[AAA] ", 0)
l.Printf("Listening 0.0.0.0%s", PORT)
l.Fatal(http.ListenAndServe(PORT, router))
If I want to do this, I have to add start := time.Now() and time.Since(start) manually to each router in my code.package main
I think I should wrap it and use interface but don't know how to get started.
How do I implement one simple logging interface and apply all the routed handlers so that I can debug with all the loggings...
Negroni does like:
router := mux.NewRouter()
router.HandleFunc("/", HomeHandler)
n := negroni.New(Middleware1, Middleware2)
// Or use a middleware with the Use() function
n.Use(Middleware3)
// router goes last
n.UseHandler(router)
n.Run(":3000")
Wrap the root handler with a handler that logs and delegates to another handler:
type RequestLogger struct {
h http.Handler
l *Logger
}
func (rl RequestLogger) ServeHTTP(w http.ResponseWriter, r *http.Request) {
start := time.Now()
rl.l.Printf("Started %s %s", r.Method, r.URL.Path)
rl.h.ServeHTTP(w, r)
rl.l.Printf("Completed %s %s in %v", r.Method, r.URL.Path, time.Since(start))
}
---
l := log.New(os.Stdout, "[AAA] ", 0)
l.Printf("Listening 0.0.0.0%s", PORT)
l.Fatal(http.ListenAndServe(PORT, RequestLogger{h:router, l:l}))
A simple middleware interceptor is probably the right approach. If you'd like an example, see a simple one here: https://github.com/jadekler/git-go-websiteskeleton/blob/master/main.go#L49. This can be contracted down to a smaller function, but YMMV.
Here is the relevant code:
At the top of your handlefuncs:
http.HandleFunc("/", httpInterceptor)
In your middleware:
func httpInterceptor(w http.ResponseWriter, req *http.Request) {
router.ServeHTTP(w, req)
logAccess(w, req)
}
Where logAccess is a function that logs whatever you'd like it to. For an example, see here.
It's rather easy to implement your own middleware in Go, one approach is something like:
var logger = log.New(os.Stdout, "[something shiny] ", 0)
func httpLogger(fn func(w http.ResponseWriter, r *http.Request)) func(w http.ResponseWriter, r *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
logger.Printf("Started %s %s", r.Method, r.URL.Path)
fn(w, r)
logger.Printf("Completed in %v", time.Since(start))
}
}
....
router.Handle("GET", "/", httpLogger(handler))

about http hijacking and keep-alive

i use
resp, err := http.Get("http://example.com/")
get a http.Response, and i want to exactly write to a http handler, but only http.ResponseWriter, so i hijack it.
...
webConn, webBuf, err := hj.Hijack()
if err != nil {
// handle error
}
defer webConn.Close()
// Write resp
resp.Write(webBuf)
...
Write raw request
But When i hijack, http connection can't reuse (keep-alive), so it slow.
How to solve?
Thanks! Sorry for my pool English.
update 12/9
keep-alive, It keep two tcp connection, and can reuse.
but when i hijack, and conn.Close(), It can't reuse old connection, so it create a new tcp connection when i each refresh.
Do not use hijack, Because once hijack, the HTTP server library will not do anything else with the connection, So can't reuse.
I change way, copy Header and Body, look like reverse proxy (http://golang.org/src/pkg/net/http/httputil/reverseproxy.go), Is works.
Example:
func copyHeader(dst, src http.Header) {
for k, w := range src {
for _, v := range w {
dst.Add(k, v)
}
}
}
func copyResponse(r *http.Response, w http.ResponseWriter) {
copyHeader(w.Header(), r.Header)
w.WriteHeader(r.StatusCode)
io.Copy(w, r.Body)
}
func handler(w http.ResponseWriter, r *http.Response) {
resp, err := http.Get("http://www.example.com")
if err != nil {
// handle error
}
copyResponse(resp, w)
}
It seem that once the connection is closed the keep-alive connection closes as well.
One possible solution would be to prevent the connection from closing until desired, but I'm not sure if that good advise.
Maybe the correct solution involves creating a instance of net.TCPConn, copying the connection over it, then calling .SetKeepAlive(true).
Before running the below example, launch another terminal with netstat -antc | grep 9090.
Routes in example:
localhost:9090/ok is a basic (non-hijacked) connection
localhost:9090 is a hijacked connection, lasting for 10 seconds.
Example
package main
import (
"fmt"
"net/http"
"sync"
"time"
)
func checkError(e error) {
if e != nil {
panic(e)
}
}
var ka_seconds = 10
var conn_id = 0
func main() {
http.HandleFunc("/ok", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "ok")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
conn_id++
fmt.Printf("Connection %v: Keep-alive is enabled %v seconds\n", conn_id, ka_seconds)
hj, ok := w.(http.Hijacker)
if !ok {
http.Error(w, "webserver doesn't support hijacking", http.StatusInternalServerError)
return
}
conn, bufrw, err := hj.Hijack()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Don't forget to close the connection:
time.AfterFunc(time.Second* time.Duration(ka_seconds), func() {
conn.Close()
fmt.Printf("Connection %v: Keep-alive is disabled.\n", conn_id)
})
resp, err := http.Get("http://www.example.com")
checkError(err)
resp.Write(bufrw)
bufrw.Flush()
})
fmt.Println("Listing to localhost:9090")
http.ListenAndServe(":9090", nil)
}
Related issue: http://code.google.com/p/go/issues/detail?id=5645

Resources