I believe the problem lies in the url values. When I post this to the server, I will a 400 Bad Request: telling me that I need to have an email value. This leads me to believe that either the email value in editForm is getting parsed incorrectly, or the the first_value is, and then "tainting" the rest. I have seen this: Make a URL-encoded POST request using `http.NewRequest(...)` and believe I am doing everything right, but this is throwing me off.
editForm := url.Values{}
editForm.Add("first_name", "supercool")
editForm.Add("email", "wow#example.com")
editForm.Add("username", "foo")
req, err := http.NewRequest(http.MethodPost, urlEndpoint, strings.NewReader(editForm.Encode()))
if err != nil {
log.Fatalln(err)
}
client := http.Client{}
resp, err := client.Do(req)
I have double checked what the form data is supposed to be called, and I cannot see an error. For reference, this python code will work.
cn = {
"first_name": "supercool",
"email": "wow#example.com",
"username": "foo"
}
r = requests.post(urlEndpoint, data = cn)
You are not sending the content negotiation header.
Content Type
The Content-Type header field specifies the nature of the data in the
body of an entity by giving media type and subtype identifiers, and by providing auxiliary information that may be required for certain
media types. After the media type and subtype names, the remainder of the header field is simply a set of parameters, specified in an attribute=value notation. The ordering of parameters is not significant.
Here in this case the content is encoded in application/x-www-form-urlencoded so it has to be communicated to the server using Content-Type header
Please add the following before sending the request
r.Header.Add("Content-Type", "application/x-www-form-urlencoded")
Related
I'm developing a Security Scanner and therefore need to send HTTP requests which don't honor RFC specifications. However, golang is very strict to comply with these.
Issue
I want to send a HTTP request which contains prohibited special characters such as "".
For example: "Ill\egal": "header value"
However, golang always throws the error: 'net/http: invalid header field name "Ill\egal"'.
This error is thrown on line 523 at https://go.dev/src/net/http/transport.go
Issue
I want to send a single HTTP request which contains either two content-length, two transfer-encoding or one content-length & one transfer-encoding header (for HTTP request smuggling). Those need sometimes to have wrong values.
However, it isn't possible to set those headers oneself, they are generated automatically. So it's only possible to use one of these headers with a correct value.
I've bypassed this by using a Raw TCP Stream, however this solution isn't satisfying, as I can't use a proxy this way: Use Dialer with Proxy. Route TCP stream through Proxy
Issue
I want to send a HTTP request where the header name is mixed upper and lowercase. E.g. "HeAdErNaMe": "header value".
This is possible for HTTP 1 requests by writing directly to the header map (req.Header["HeAdErNaMe"] = []string{"header value"})
However for HTTP 2 requests the headers will still be capitalized to meet the RFC specifications.
You can dump request into a buffer, modify the buffer (with regexp or replace), and send modified buffer to the host using net.Dial.
Example:
package main
import (
"bufio"
"crypto/tls"
"fmt"
"log"
"net/http"
"net/http/httputil"
"strings"
)
func main() {
// create and dump request
req, err := http.NewRequest(http.MethodGet, "https://golang.org", nil)
if err != nil {
log.Fatal(err)
}
req.Header.Add("User-Agent", "aaaaa")
buf, err := httputil.DumpRequest(req, true)
if err != nil {
log.Fatal(err)
}
// Corrupt request
str := string(buf)
str = strings.Replace(str, "User-Agent: aaaaa", "UsEr-AgEnT: aaa\"aaa", 1)
println(str)
// Dial and send raw request text
conn, err := tls.Dial("tcp", "golang.org:443", nil)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
fmt.Fprintf(conn, str)
// Read response
br := bufio.NewReader(conn)
resp, err := http.ReadResponse(br, nil)
if err != nil {
log.Fatal(err)
}
log.Printf("%+v", resp)
}
Below handler handles GET request, without populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Below handler handles GET request, populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
rw.Header().Add("Content-Type", "application/json")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Both cases are working fine with simple curl request.
For any http client,
When do we need to populate content-type header, before sending the response, to client?
Always read the documentation first!
The answer to this is clearly covered here (emphasis obviously added):
// If WriteHeader has not yet been called, Write calls
// WriteHeader(http.StatusOK) before writing the data. If the Header
// does not contain a Content-Type line, Write adds a Content-Type set
// to the result of passing the initial 512 bytes of written data to
// DetectContentType. Additionally, if the total size of all written
// data is under a few KB and there are no Flush calls, the
// Content-Length header is added automatically.
To explicitly answer your secondary question:
When do we need to populate content-type header?
Any time you don't want it to be automatically detected. Automatic detection is imprecise, so you generally don't want to rely on it.
I am using Go to make an HTTP GET request to an external web service. For some reason, the body of the response is always empty; the content length is always zero bytes. The response status code is always 200, however, and the call to Client.Do returns no error. The request requires an Authorization header, so I am using the http.NewRequest / http.Client.Do pattern to submit the request, as you'll see below. I have done requests similar to these in the past, but never using a GET that required a header. It seems unlikely that this the cause, but I wonder if it may be related. If anyone can spot any potential issues with the pattern used or perhaps has had a similar experience, I'd really appreciate any help.
Thank you.
if req, err := http.NewRequest("GET", "https://api.molt.in/v1/orders/11111111/items", nil); err != nil {
return nil, err
} else {
client := &http.Client{}
req.Header.Add("Authorization", "secretToken")
if resp, err := client.Do(req); err != nil {
return nil, err
} else {
defer resp.Body.Close()
return readBody(resp.Body)
}
}
I finally discovered the source of the problem. It had nothing to do with the request being made, or the response being received. It had to do with the parsing of the response.
I was using bufio.NewScanner.Text to attempt to convert the response body into a string. Replacing this call with one to ioutil.ReadAll output the string that I originally expected.
Thanks for all of your help, and apologies for the misleading question.
I'm trying to send a GET request to the following api:
https://poloniex.com/public?command=returnOrderBook
w/ URL parameters:
currencyPair=BTC_ETH
depth=20
--> ¤cyPair=BTC_ETH&depth=20
I try to setup and execute my request as so: (note I've removed error checking for brevity)
pair := "BTC_ETH"
depth := 20
reqURL := "https://poloniex.com/public?command=returnOrderBook"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
fmt.Printf("\n Values = %s\n", values.Encode()) //DEBUG
req, err := http.NewRequest("GET", reqURL, strings.NewReader(values.Encode()))
fmt.Printf("\nREQUEST = %+v\n", req) //DEBUG
resp, err := api.client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Printf("\nREST CALL RETURNED: %X\n",body) //DEBUG
My DEBUG print statements print out the following:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:{Reader:0xc82028e840} ContentLength:29 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: {"error":"Please specify a currency pair."}
Playing around with Postman I figured out the API only returns this error when the currencyPair parameter is not specified (including miscapitalized). I can't figure out why the request doesn't include the URL parameters I specified as it's obvious from my debug print statements that the values.Encode() is correct. The content length in the request corresponds to the right amount of chars (bytes) needed for URL parameters.
Now after playing around a bit I found a solution.
If I replace the http.NewRequest() line with the following it works:
req, err := http.NewRequest(HTTPType, reqURL + "&" + values.Encode(), nil)
However, it's really bothering me why the original statement doesn't work.
The new DEBUG output is:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook¤cyPair=BTC_ETH&depth=5 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: *way too long, just assume it's the correct financial data*
Would love some input on what I did wrong in the original statement. I used the same method (original) for a different api endpoint w/ URL parameters and it worked fine. Confused on why it didn't work in this case.
GET requests should not contain a body. Instead, you need to put the form into the query string.
Here's the proper way to do that, without hacky string concatenation:
reqURL := "https://poloniex.com/public"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
values.Set("command", "returnOrderBook")
uri, _ := url.Parse(reqURL)
uri.Query = values.Encode()
reqURL = uri.String()
fmt.Println(reqURL)
req, err := http.NewRequest("GET", reqURL, nil)
if err != nil {
panic(err) // NewRequest only errors on bad methods or un-parsable urls
}
https://play.golang.org/p/ZCLUu7UgZL
Using go, how can I parse the Content-Disposition header retrieved from an http HEAD request to obtain the filename of the file?
Additionally, how do I retrieve the header itself from the http HEAD response? Is something like this correct?
resp, err := http.Head("http://example.com/")
//handle error
contentDisposition := resp.Header.Get("Content-Disposition")
The mime/multipart package specifies a method on the Part type that returns the filename (called FileName), but it's not clear to me how I should construct a Part, or from what.
You can parse the Content-Disposition header using the mime.ParseMediaType function.
disposition, params, err := mime.ParseMediaType(`attachment;filename="foo.png"`)
filename := params["filename"] // set to "foo.png"
This will also work for Unicode file names in the header (e.g. Content-Disposition: attachment;filename*="UTF-8''fo%c3%b6.png").
You can experiment with this here: http://play.golang.org/p/AjWbJB8vUk