I'm trying to extract a value from an HTTP POST request body (in my simple Go HTTP server) using the net/http PostFormValue and my output is an empty string when I'm looking for the any key in general, but in my case trying to fetch the hub.secret for use in a HMAC check. I use Postman to send the request to my localhost:8080 instance using the Gorilla/mux router, with header Content-Type: application/x-www-form-urlencoded set.
My handler looks like so:
func rootPostHandler(w http.ResponseWriter, r *http.Request) {
var expectedMac []byte
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("r.Body is:", string(body)) // debug: print the request POST body
message := body // debug: set message just for extra clarity
errParse := r.ParseForm()
if errParse != nil {
// handle err
}
secret := []byte(r.PostFormValue("hub.secret"))
log.Println("secret is: ", string(secret))
mac := hmac.New(sha256.New, secret)
mac.Write(message)
expectedMac = mac.Sum(nil)
fmt.Println("Is HMAC equal? ", hmac.Equal(message, expectedMac))
w.Header().Add("X-Hub-Signature", "sha256="+string(message))
}
The r.Body:
hub.callback=http%253A%252F%252Fweb-sub-client%253A8080%252FbRxvcmOcNk&hub.mode=subscribe&hub.secret=xTgSGLOtPNrBLLgYcKnL&hub.topic=%252Fa%252Ftopic
And the output for print secret etc is empty string, meaning it can't find hub.secret, right? What am I missing here?
The application reads the request body to EOF on this line:
body, err := ioutil.ReadAll(r.Body)
ParseForm returns an empty form because the body is at EOF at this line:
errParse := r.ParseForm()
The request body is read from the network connection. The request body cannot be read a second time.
Remove the call to ioutil.ReadAll or create a new body reader using the data returned from ioutil.ReadAll:
r.Body = io.NopCloser(bytes.NewReader(body))
Related
I'm developing a Security Scanner and therefore need to send HTTP requests which don't honor RFC specifications. However, golang is very strict to comply with these.
Issue
I want to send a HTTP request which contains prohibited special characters such as "".
For example: "Ill\egal": "header value"
However, golang always throws the error: 'net/http: invalid header field name "Ill\egal"'.
This error is thrown on line 523 at https://go.dev/src/net/http/transport.go
Issue
I want to send a single HTTP request which contains either two content-length, two transfer-encoding or one content-length & one transfer-encoding header (for HTTP request smuggling). Those need sometimes to have wrong values.
However, it isn't possible to set those headers oneself, they are generated automatically. So it's only possible to use one of these headers with a correct value.
I've bypassed this by using a Raw TCP Stream, however this solution isn't satisfying, as I can't use a proxy this way: Use Dialer with Proxy. Route TCP stream through Proxy
Issue
I want to send a HTTP request where the header name is mixed upper and lowercase. E.g. "HeAdErNaMe": "header value".
This is possible for HTTP 1 requests by writing directly to the header map (req.Header["HeAdErNaMe"] = []string{"header value"})
However for HTTP 2 requests the headers will still be capitalized to meet the RFC specifications.
You can dump request into a buffer, modify the buffer (with regexp or replace), and send modified buffer to the host using net.Dial.
Example:
package main
import (
"bufio"
"crypto/tls"
"fmt"
"log"
"net/http"
"net/http/httputil"
"strings"
)
func main() {
// create and dump request
req, err := http.NewRequest(http.MethodGet, "https://golang.org", nil)
if err != nil {
log.Fatal(err)
}
req.Header.Add("User-Agent", "aaaaa")
buf, err := httputil.DumpRequest(req, true)
if err != nil {
log.Fatal(err)
}
// Corrupt request
str := string(buf)
str = strings.Replace(str, "User-Agent: aaaaa", "UsEr-AgEnT: aaa\"aaa", 1)
println(str)
// Dial and send raw request text
conn, err := tls.Dial("tcp", "golang.org:443", nil)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
fmt.Fprintf(conn, str)
// Read response
br := bufio.NewReader(conn)
resp, err := http.ReadResponse(br, nil)
if err != nil {
log.Fatal(err)
}
log.Printf("%+v", resp)
}
My goal is to scrape a website that requires me to log in first using HTTP requests in Golang. I actually succeeded by finding out I can send a post request to the website writing form-data into the body of the request. When I test this through an API development software I use called Postman, the response is instantaneous with no delays. However, when performing the request with an HTTP client in Go, there is a consistent 60 second delay every single time. I end up getting a logged in page, but for my program I need the response to be nearly instantaneous.
As you can see in my code, I've tried adding a bunch of headers to the request like "Connection", "Content-Type", "User-Agent" since I thought maaaaaybe the website can tell I'm requesting from a program and is forcing me to wait 60 seconds for a response. Adding these headers to make my request more legitimate(?) doesn't work at all.
Is the delay coming from Go's HTTP client being slow or is there something wrong with how I'm forming my HTTP POST request? Also, was I on to something with my headers and HTTP client is rewriting them when they send out?
Here's my simple program...
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
"net/http/cookiejar"
"os"
)
func main() {
url := "https://easypronunciation.com/en/log-in"
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
_ = writer.WriteField("email", "foo#bar.com")
_ = writer.WriteField("password", "*********")
_ = writer.WriteField("persistent_login", "on")
_ = writer.WriteField("submit", "")
err := writer.Close()
if err != nil {
fmt.Println(err)
}
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", writer.FormDataContentType())
req.Header.Set("Connection", "Keep-Alive")
req.Header.Set("Accept-Language", "en-US")
req.Header.Set("User-Agent", "Mozilla/5.0")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
f, err := os.Create("response.html")
defer f.Close()
res.Write(f)
}
I doubt, this is the go client library too. I would suggest printing out the latencies for different components and see if/where the 60 second delay is. I would also replace and try different URLs instead
Below handler handles GET request, without populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Below handler handles GET request, populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
rw.Header().Add("Content-Type", "application/json")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Both cases are working fine with simple curl request.
For any http client,
When do we need to populate content-type header, before sending the response, to client?
Always read the documentation first!
The answer to this is clearly covered here (emphasis obviously added):
// If WriteHeader has not yet been called, Write calls
// WriteHeader(http.StatusOK) before writing the data. If the Header
// does not contain a Content-Type line, Write adds a Content-Type set
// to the result of passing the initial 512 bytes of written data to
// DetectContentType. Additionally, if the total size of all written
// data is under a few KB and there are no Flush calls, the
// Content-Length header is added automatically.
To explicitly answer your secondary question:
When do we need to populate content-type header?
Any time you don't want it to be automatically detected. Automatic detection is imprecise, so you generally don't want to rely on it.
I'm trying to send a GET request to the following api:
https://poloniex.com/public?command=returnOrderBook
w/ URL parameters:
currencyPair=BTC_ETH
depth=20
--> ¤cyPair=BTC_ETH&depth=20
I try to setup and execute my request as so: (note I've removed error checking for brevity)
pair := "BTC_ETH"
depth := 20
reqURL := "https://poloniex.com/public?command=returnOrderBook"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
fmt.Printf("\n Values = %s\n", values.Encode()) //DEBUG
req, err := http.NewRequest("GET", reqURL, strings.NewReader(values.Encode()))
fmt.Printf("\nREQUEST = %+v\n", req) //DEBUG
resp, err := api.client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Printf("\nREST CALL RETURNED: %X\n",body) //DEBUG
My DEBUG print statements print out the following:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:{Reader:0xc82028e840} ContentLength:29 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: {"error":"Please specify a currency pair."}
Playing around with Postman I figured out the API only returns this error when the currencyPair parameter is not specified (including miscapitalized). I can't figure out why the request doesn't include the URL parameters I specified as it's obvious from my debug print statements that the values.Encode() is correct. The content length in the request corresponds to the right amount of chars (bytes) needed for URL parameters.
Now after playing around a bit I found a solution.
If I replace the http.NewRequest() line with the following it works:
req, err := http.NewRequest(HTTPType, reqURL + "&" + values.Encode(), nil)
However, it's really bothering me why the original statement doesn't work.
The new DEBUG output is:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook¤cyPair=BTC_ETH&depth=5 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: *way too long, just assume it's the correct financial data*
Would love some input on what I did wrong in the original statement. I used the same method (original) for a different api endpoint w/ URL parameters and it worked fine. Confused on why it didn't work in this case.
GET requests should not contain a body. Instead, you need to put the form into the query string.
Here's the proper way to do that, without hacky string concatenation:
reqURL := "https://poloniex.com/public"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
values.Set("command", "returnOrderBook")
uri, _ := url.Parse(reqURL)
uri.Query = values.Encode()
reqURL = uri.String()
fmt.Println(reqURL)
req, err := http.NewRequest("GET", reqURL, nil)
if err != nil {
panic(err) // NewRequest only errors on bad methods or un-parsable urls
}
https://play.golang.org/p/ZCLUu7UgZL
From the documentation it states that
For server requests the Request Body is always non-nil but will return EOF immediately when no body is present.
For ContentLength, the documentation states
For client requests, a value of 0 means unknown if Body is not nil.
So is it better to check for ContentLength
r *http.Request
if r.ContentLength == 0 {
//empty body
}
or to check EOF
type Input struct {
Name *string `json:"name"`
}
input := new(Input)
if err := json.NewDecoder(r.Body).Decode(input); err.Error() == "EOF" {
//empty body
}
You always need to read the body to know what the contents are. The client could send the body in chunked encoding with no Content-Length, or it could even have an error and send a Content-Length and no body. The client is never obligated to send what it says it's going to send.
The EOF check can work if you're only checking for the empty body, but I would still also check for other error cases besides the EOF string.
err := json.NewDecoder(r.Body).Decode(input)
switch {
case err == io.EOF:
// empty body
case err != nil:
// other error
}
You can also read the entire body before unmarshalling:
body, err := ioutil.ReadAll(r.Body)
or if you're worried about too much data
body, err := ioutil.ReadAll(io.LimitReader(r.Body, readLimit))
if http.Request().Body == http.NoBody {
// TODO.
}