I want to make an http/2 request in go and got to a few issues there.
How to make proper http/2 requests in go?
Error: Get "https://webhook.site/aae1e0ab-3e48-49c8-8cd0-526e12ee4077": http2: unexpected ALPN protocol ""; want "h2" (Why? Other sites are working)
Code:
t := &http2.Transport{}
c := &http.Client{
Transport: t,
}
r, err := http.NewRequest("GET", "https://webhook.site/aae1e0ab-3e48-49c8-8cd0-526e12ee4077", nil)
if err != nil {
fmt.Println(err)
}
r.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.2 Safari/605.1.15")
r.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
resp, err := c.Do(r)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(body))
```
As was mentioned, that server doesn't support HTTP2:
PS C:\> curl.exe -I --http2-prior-knowledge https://webhook.site
curl: (16) Error in the HTTP2 framing layer
Contrast with one that does:
PS C:\> curl.exe -I --http2-prior-knowledge https://example.com
HTTP/2 200
https://curl.se/docs/manpage.html#--http2-prior-knowledge
Related
I am trying to read from a tcp connection which contains HTTP/2 data. Below is the code for reading HEADERS frame -
framer := http2.NewFramer(conn, conn)
frame, _ := framer.ReadFrame()
fmt.Printf("fh type: %s\n", frame.Header().Type)
fmt.Printf("fh type: %d\n", frame.Header().Type)
fmt.Printf("fh flag: %d\n", frame.Header().Flags)
fmt.Printf("fh length: %d\n", frame.Header().Length)
fmt.Printf("fh streamid: %d\n", frame.Header().StreamID)
headersframe := (frame1.(*http2.HeadersFrame))
fmt.Printf("stream ended? %v\n", headersframe.StreamEnded())
fmt.Printf("block fragment: %x\n", headersframe.HeaderBlockFragment())
I send request using curl as -
curl -v https://127.0.0.1:8000/ -k --http2
This is the output I get (after reading connection preface and SETTINGS), if I read from the conn using above code -
fh type: HEADERS
fh type: 1
fh flag: 5
fh length: 30
fh streamid: 1
stream ended? true
block fragment: 828487418a089d5c0b8170dc6c4d8b7a8825b650c3abb6f2e053032a2f2a
I understand the ouput, except the block fragment part and how to decode it into ascii string? I want to know the protocol/method/url path information.
The "header block fragment" is encoded using HPACK.
Go has an implementation to encode and decode HPACK, so you don't have to write your own.
You can find here an example of using both the encoder and decoder Go API.
I figured it out using Go hpack library (https://godoc.org/golang.org/x/net/http2/hpack) -
decoder := hpack.NewDecoder(2048, nil)
hf, _ := decoder.DecodeFull(headersframe.HeaderBlockFragment())
for _, h := range hf {
fmt.Printf("%s\n", h.Name + ":" + h.Value)
}
This prints -
:method:GET
:path:/
:scheme:https
:authority:127.0.0.1:5252
user-agent:curl/7.58.0
accept:*/*
I have an issue with Go's http.Server, which I'm embedding in a struct that is supposed to control the server startup and shutdown. The struct looks like this:
type HTTPListen struct {
Consumers []pipeline.Consumer
Cfg HTTPListenConfig
Srv *http.Server
Logger log.Logger
wg *sync.WaitGroup
mu sync.Mutex
state State
}
The issue is that in my test code, I call my struct's Start() method (which in turn runs the Serve() method on the http.Server), check a few vars, and then call Stop(), whitch Shutdown()s the server and then waits for the http.Server to exit (return err from the Serve() method).
Now, for some reason, the Serve() method seems to just hang on the WaitGroup.Wait(), when I try to shutdown the server immediately after starting. When I add a short pause (tried 100ms), or when running the tests with the race detector, It works just fine.
Not sure if it matters, but there are no incoming requests between calling Serve() and Shutdown().
EDIT: link to a playground minimal example. If you comment out the time.Sleep call, the program hangs.
Here is the relevant code for the two methods:
func (h *HTTPListen) Start() error {
h.Logger.Log("msg", "starting HTTPListen input")
addr := h.Cfg.ListenAddr
ln, err := net.Listen("tcp", addr)
if err != nil {
h.Logger.Log("msg", "failed to create listener on tcp/"+addr+": "+err.Error())
h.setState(StateFailed)
return err
}
h.wg.Add(1)
go func() {
defer h.wg.Done()
err := h.Srv.Serve(ln)
h.Logger.Log("msg", "HTTP server stopped: "+err.Error())
}()
h.setState(StateStarted)
h.Logger.Log("msg", "HTTPListen input started")
return nil
}
Stop method:
func (h *HTTPListen) Stop() error {
h.Logger.Log("msg", "stopping HTTPListen input")
ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
if err := h.Srv.Shutdown(ctx); err != nil {
h.Logger.Log("msg", "HTTP server shutdown deadline expired")
}
h.wg.Wait()
h.setState(StateStopped)
h.Logger.Log("msg", "HTTPListen input stopped")
return nil
}
Log output:
kwz#cyclone ~/s/stblogd> go test -v ./pkg/pipeline/input/ -run TestHTTPListen_StartStop
=== RUN TestHTTPListen_StartStop
msg="starting HTTPListen input"
msg="HTTPListen input started"
msg="stopping HTTPListen input"
... hangs indefinitely
Log output when running tests with the race detector:
kwz#cyclone ~/s/stblogd> go test -race -v ./pkg/pipeline/input/ -run TestHTTPListen_StartStop
=== RUN TestHTTPListen_StartStop
msg="starting HTTPListen input"
msg="HTTPListen input started"
msg="stopping HTTPListen input"
msg="HTTP server stopped: http: Server closed"
msg="HTTPListen input stopped"
--- PASS: TestHTTPListen_StartStop (0.00s)
PASS
ok stblogd/pkg/pipeline/input 1.007s
I'm tempted to just slap a short delay on the test and call it a day, but I would like to know why it behaves like this.
This is a known issue, see this thread:
https://github.com/golang/go/issues/20239
Hopefully they will fix it soon but for now it sounds like adding a short delay in your test is the simplest solution - it probably doesn't come up in real world use much because you won't trigger a shutdown so soon after starting.
I run:
curl -i -X POST -H "Content-Type: application/octet-stream" --data #race.mov "http://127.0.0.1:8080/api/v1/put_file"
where race.mov is 4134329 bytes. But only 1931871 are received by the program listening on port 8080.
so I try another file that's 2482905 bytes. But only 1150635 are received. Isn't that weird?
irb(main):002:0> 1931871.0 / 4134329.0
=> 0.4672755845023461
irb(main):003:0> 1150635.0 / 2482905.0
=> 0.4634228856923644
What's going on to make curl only send 46% of the binary data?
Update this same go program:
dat, err := ioutil.ReadFile("/Users/aa/Movies/race.mov")
fmt.Println(err)
endpoint := fmt.Sprintf("http://127.0.0.1:8080/api/v1/put_file")
fmt.Println(endpoint)
buffer_reader := bytes.NewReader(dat)
resp, err := http.Post(endpoint, "application/octet-stream", buffer_reader)
fmt.Println(resp, err)
makes the correct length happen so it MUST be curl
I am using the gopacket package and every time I have a TCP packet I want to check if the payload contains an HTTP request. Is there an easy way to do that instead of writing my own parser? There is also a function (see: func ReadRequest(b *bufio.Reader)) which returns a Request struct but I do not know what kind of input I should use. tcp.Payload is the byte[] array that seems to have the information I need to parse (see the following example):
// Get the TCP layer from this packet
if tcpLayer := packet.Layer(layers.LayerTypeTCP); tcpLayer != nil {
fmt.Printf("TCP ")
// Get actual TCP data from this layer
tcp, _ := tcpLayer.(*layers.TCP)
srcPort = tcp.SrcPort
dstPort = tcp.DstPort
if tcp.SYN {
fmt.Print("[SYN] ")
}
if tcp.ACK {
fmt.Print("[ACK] ")
}
if tcp.FIN {
fmt.Print("[FIN] ")
}
fmt.Printf("%s:%d > %s:%d ", srcIP, srcPort, dstIP, dstPort)
fmt.Println(string(tcp.Payload))
}
After sending an HTTP request I get the following output:
PKT [001] TCP [SYN] 192.168.2.6:59095 > 192.168.3.5:80
PKT [002] TCP [SYN] [ACK] 192.168.3.5:80 > 192.168.2.6:59095
PKT [003] TCP [ACK] 192.168.2.6:59095 > 192.168.3.5:80
PKT [004] TCP [ACK] 192.168.2.6:59095 > 192.168.3.5:80 GET /certificates/test.pdf HTTP/1.1
User-Agent: Wget/1.15 (linux-gnu)
Accept: */*
Host: 192.168.3.5
Connection: Keep-Alive
Any suggestions are welcome...
With help from this question: How to parse http headers in Go the following attempt with http.ReadRequest() works...
if tcpLayer := packet.Layer(layers.LayerTypeTCP); tcpLayer != nil {
tcp, _ := tcpLayer.(*layers.TCP)
if len(tcp.Payload) != 0 {
reader := bufio.NewReader(bytes.NewReader(tcp.Payload))
httpReq, err := http.ReadRequest(reader)
...
}
}
I got this panic:
C:\Users\loow\Desktop\USBWebserver v8.5\duplicate_submissions>go run server.go
2015/10/23 13:00:39 http: panic serving [::1]:63867: runtime error: invalid memo
ry address or nil pointer dereference
goroutine 5 [running]:
net/http.(*conn).serve.func1(0xc0820a1810, 0x3b55b8, 0xc082024040)
c:/go/src/net/http/server.go:1287 +0xbc
main.login(0x2990058, 0xc0820d4000, 0xc0820be1c0)
C:/Users/loow/Desktop/USBWebserver v8.5/duplicate_submissions/server.go:
27 +0x5a5
net/http.HandlerFunc.ServeHTTP(0x8326a8, 0x2990058, 0xc0820d4000, 0xc0820be1c0)
c:/go/src/net/http/server.go:1422 +0x41
net/http.(*ServeMux).ServeHTTP(0xc082062360, 0x2990058, 0xc0820d4000, 0xc0820be1
c0)
c:/go/src/net/http/server.go:1699 +0x184
net/http.serverHandler.ServeHTTP(0xc08200c360, 0x2990058, 0xc0820d4000, 0xc0820b
e1c0)
c:/go/src/net/http/server.go:1862 +0x1a5
net/http.(*conn).serve(0xc0820a1810)
c:/go/src/net/http/server.go:1361 +0xbf5
created by net/http.(*Server).Serve
c:/go/src/net/http/server.go:1910 +0x3fd
2015/10/23 13:00:39 http: panic serving [::1]:63868: runtime error: invalid memo
ry address or nil pointer dereference
goroutine 33 [running]:
net/http.(*conn).serve.func1(0xc082114000, 0x3b55b8, 0xc082112000)
c:/go/src/net/http/server.go:1287 +0xbc
main.login(0x2990058, 0xc0821140b0, 0xc0821200e0)
C:/Users/loow/Desktop/USBWebserver v8.5/duplicate_submissions/server.go:
27 +0x5a5
net/http.HandlerFunc.ServeHTTP(0x8326a8, 0x2990058, 0xc0821140b0, 0xc0821200e0)
c:/go/src/net/http/server.go:1422 +0x41
net/http.(*ServeMux).ServeHTTP(0xc082062360, 0x2990058, 0xc0821140b0, 0xc0821200
e0)
c:/go/src/net/http/server.go:1699 +0x184
net/http.serverHandler.ServeHTTP(0xc08200c360, 0x2990058, 0xc0821140b0, 0xc08212
00e0)
c:/go/src/net/http/server.go:1862 +0x1a5
net/http.(*conn).serve(0xc082114000)
c:/go/src/net/http/server.go:1361 +0xbf5
created by net/http.(*Server).Serve
c:/go/src/net/http/server.go:1910 +0x3fd
exit status 2
Whit this code:
package main
import(
"fmt"
"net/http"
"html/template"
"log"
"time"
"crypto/md5"
"io"
"strconv"
)
func loginForm(w http.ResponseWriter, r *http.Request){
}
func login(w http.ResponseWriter, r *http.Request){
fmt.Println(r.Method)
if r.Method == "GET"{
cruTime := time.Now().Unix()
h := md5.New()
io.WriteString(h,strconv.FormatInt(cruTime,10))
token := fmt.Sprintf("%x", h.Sum(nil))
fmt.Println(token)
t, err := template.ParseFiles("templates/index.gtpl")
fmt.Println(err.Error())
err = t.Execute(w, token)
fmt.Println(err.Error())
} else{
r.ParseForm()
token := r.Form.Get("token")
if token != ""{
fmt.Println(token)
} else{
fmt.Println("There is no token")
}
fmt.Println("username length: ", len(r.Form["username"][0]))
fmt.Println("username: ", template.HTMLEscapeString(r.Form.Get("username")))
fmt.Println("password: ", template.HTMLEscapeString(r.Form.Get("password")))
template.HTMLEscape(w, []byte(r.Form.Get("username")))
}
}
func main(){
http.HandleFunc("/", loginForm)
http.HandleFunc("/login", login)
err := http.ListenAndServe(":9090", nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
I cant fix it, I tried everything what I found in the stackoverflow. What is the problem? There is no error, and the panic said that the problem in t, err := template.ParseFiles("templates/index.gtpl")..
There is the template file:
<input type="checkbox" name="interest" value="football">Football
<input type="checkbox" name="interest" value="basketball">Basketball
<input type="checkbox" name="interest" value="tennis">Tennis
Username: <input type="text" name="username">
Password: <input type="password" name="password">
<input type="hidden" name="token" value="{{.}}">
<input type="submit" value="Login">
The panic stacktrace gives you this info :
2015/10/23 13:00:39 http: panic serving [::1]:63868: runtime error: invalid memory address or nil pointer dereference goroutine 33 [running]:
It means you're trying to access something that does not exist (nil pointer).
Then the first line that comes from your file is this one :
v8.5/duplicate_submissions/server.go:27
Which is there :
26: t, err := template.ParseFiles("templates/index.gtpl")
27: fmt.Println(err.Error())
28: err = t.Execute(w, token)
It means err is nil.
Solution
If you get the error, you cannot continue the process. That's the reason why you cannot just print out the error. In order to stop gracefully the process, you need to return an HTTP status code and then return. For the case above, you can return a code 500 (internal server error).
t, err := template.ParseFiles("templates/index.gtpl")
if err != nil {
fmt.Println(err) // Ugly debug output
w.WriteHeader(http.StatusInternalServerError) // Proper HTTP response
return
}
That has to be done for a template.ParseFiles and t.Execute too.
By the way, that is called the "comma ok" pattern