Golang 'http.NewRequest(method, url, body)' fails to create correctly formatted request - http

I'm trying to send a GET request to the following api:
https://poloniex.com/public?command=returnOrderBook
w/ URL parameters:
currencyPair=BTC_ETH
depth=20
--> &currencyPair=BTC_ETH&depth=20
I try to setup and execute my request as so: (note I've removed error checking for brevity)
pair := "BTC_ETH"
depth := 20
reqURL := "https://poloniex.com/public?command=returnOrderBook"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
fmt.Printf("\n Values = %s\n", values.Encode()) //DEBUG
req, err := http.NewRequest("GET", reqURL, strings.NewReader(values.Encode()))
fmt.Printf("\nREQUEST = %+v\n", req) //DEBUG
resp, err := api.client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Printf("\nREST CALL RETURNED: %X\n",body) //DEBUG
My DEBUG print statements print out the following:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:{Reader:0xc82028e840} ContentLength:29 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: {"error":"Please specify a currency pair."}
Playing around with Postman I figured out the API only returns this error when the currencyPair parameter is not specified (including miscapitalized). I can't figure out why the request doesn't include the URL parameters I specified as it's obvious from my debug print statements that the values.Encode() is correct. The content length in the request corresponds to the right amount of chars (bytes) needed for URL parameters.
Now after playing around a bit I found a solution.
If I replace the http.NewRequest() line with the following it works:
req, err := http.NewRequest(HTTPType, reqURL + "&" + values.Encode(), nil)
However, it's really bothering me why the original statement doesn't work.
The new DEBUG output is:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook&currencyPair=BTC_ETH&depth=5 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: *way too long, just assume it's the correct financial data*
Would love some input on what I did wrong in the original statement. I used the same method (original) for a different api endpoint w/ URL parameters and it worked fine. Confused on why it didn't work in this case.

GET requests should not contain a body. Instead, you need to put the form into the query string.
Here's the proper way to do that, without hacky string concatenation:
reqURL := "https://poloniex.com/public"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
values.Set("command", "returnOrderBook")
uri, _ := url.Parse(reqURL)
uri.Query = values.Encode()
reqURL = uri.String()
fmt.Println(reqURL)
req, err := http.NewRequest("GET", reqURL, nil)
if err != nil {
panic(err) // NewRequest only errors on bad methods or un-parsable urls
}
https://play.golang.org/p/ZCLUu7UgZL

Related

Unable to extract value from r.PostFormValue in Go?

I'm trying to extract a value from an HTTP POST request body (in my simple Go HTTP server) using the net/http PostFormValue and my output is an empty string when I'm looking for the any key in general, but in my case trying to fetch the hub.secret for use in a HMAC check. I use Postman to send the request to my localhost:8080 instance using the Gorilla/mux router, with header Content-Type: application/x-www-form-urlencoded set.
My handler looks like so:
func rootPostHandler(w http.ResponseWriter, r *http.Request) {
var expectedMac []byte
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("r.Body is:", string(body)) // debug: print the request POST body
message := body // debug: set message just for extra clarity
errParse := r.ParseForm()
if errParse != nil {
// handle err
}
secret := []byte(r.PostFormValue("hub.secret"))
log.Println("secret is: ", string(secret))
mac := hmac.New(sha256.New, secret)
mac.Write(message)
expectedMac = mac.Sum(nil)
fmt.Println("Is HMAC equal? ", hmac.Equal(message, expectedMac))
w.Header().Add("X-Hub-Signature", "sha256="+string(message))
}
The r.Body:
hub.callback=http%253A%252F%252Fweb-sub-client%253A8080%252FbRxvcmOcNk&hub.mode=subscribe&hub.secret=xTgSGLOtPNrBLLgYcKnL&hub.topic=%252Fa%252Ftopic
And the output for print secret etc is empty string, meaning it can't find hub.secret, right? What am I missing here?
The application reads the request body to EOF on this line:
body, err := ioutil.ReadAll(r.Body)
ParseForm returns an empty form because the body is at EOF at this line:
errParse := r.ParseForm()
The request body is read from the network connection. The request body cannot be read a second time.
Remove the call to ioutil.ReadAll or create a new body reader using the data returned from ioutil.ReadAll:
r.Body = io.NopCloser(bytes.NewReader(body))

Why is there a 60 second delay on my HTTP POST request when using a Go HTTP client?

My goal is to scrape a website that requires me to log in first using HTTP requests in Golang. I actually succeeded by finding out I can send a post request to the website writing form-data into the body of the request. When I test this through an API development software I use called Postman, the response is instantaneous with no delays. However, when performing the request with an HTTP client in Go, there is a consistent 60 second delay every single time. I end up getting a logged in page, but for my program I need the response to be nearly instantaneous.
As you can see in my code, I've tried adding a bunch of headers to the request like "Connection", "Content-Type", "User-Agent" since I thought maaaaaybe the website can tell I'm requesting from a program and is forcing me to wait 60 seconds for a response. Adding these headers to make my request more legitimate(?) doesn't work at all.
Is the delay coming from Go's HTTP client being slow or is there something wrong with how I'm forming my HTTP POST request? Also, was I on to something with my headers and HTTP client is rewriting them when they send out?
Here's my simple program...
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
"net/http/cookiejar"
"os"
)
func main() {
url := "https://easypronunciation.com/en/log-in"
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
_ = writer.WriteField("email", "foo#bar.com")
_ = writer.WriteField("password", "*********")
_ = writer.WriteField("persistent_login", "on")
_ = writer.WriteField("submit", "")
err := writer.Close()
if err != nil {
fmt.Println(err)
}
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", writer.FormDataContentType())
req.Header.Set("Connection", "Keep-Alive")
req.Header.Set("Accept-Language", "en-US")
req.Header.Set("User-Agent", "Mozilla/5.0")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
f, err := os.Create("response.html")
defer f.Close()
res.Write(f)
}
I doubt, this is the go client library too. I would suggest printing out the latencies for different components and see if/where the 60 second delay is. I would also replace and try different URLs instead

400 Bad Request when making a custom post request

I believe the problem lies in the url values. When I post this to the server, I will a 400 Bad Request: telling me that I need to have an email value. This leads me to believe that either the email value in editForm is getting parsed incorrectly, or the the first_value is, and then "tainting" the rest. I have seen this: Make a URL-encoded POST request using `http.NewRequest(...)` and believe I am doing everything right, but this is throwing me off.
editForm := url.Values{}
editForm.Add("first_name", "supercool")
editForm.Add("email", "wow#example.com")
editForm.Add("username", "foo")
req, err := http.NewRequest(http.MethodPost, urlEndpoint, strings.NewReader(editForm.Encode()))
if err != nil {
log.Fatalln(err)
}
client := http.Client{}
resp, err := client.Do(req)
I have double checked what the form data is supposed to be called, and I cannot see an error. For reference, this python code will work.
cn = {
"first_name": "supercool",
"email": "wow#example.com",
"username": "foo"
}
r = requests.post(urlEndpoint, data = cn)
You are not sending the content negotiation header.
Content Type
The Content-Type header field specifies the nature of the data in the
body of an entity by giving media type and subtype identifiers, and by providing auxiliary information that may be required for certain
media types. After the media type and subtype names, the remainder of the header field is simply a set of parameters, specified in an attribute=value notation. The ordering of parameters is not significant.
Here in this case the content is encoded in application/x-www-form-urlencoded so it has to be communicated to the server using Content-Type header
Please add the following before sending the request
r.Header.Add("Content-Type", "application/x-www-form-urlencoded")

Empty HTTP Response Using http.Client.Do in Golang

I am using Go to make an HTTP GET request to an external web service. For some reason, the body of the response is always empty; the content length is always zero bytes. The response status code is always 200, however, and the call to Client.Do returns no error. The request requires an Authorization header, so I am using the http.NewRequest / http.Client.Do pattern to submit the request, as you'll see below. I have done requests similar to these in the past, but never using a GET that required a header. It seems unlikely that this the cause, but I wonder if it may be related. If anyone can spot any potential issues with the pattern used or perhaps has had a similar experience, I'd really appreciate any help.
Thank you.
if req, err := http.NewRequest("GET", "https://api.molt.in/v1/orders/11111111/items", nil); err != nil {
return nil, err
} else {
client := &http.Client{}
req.Header.Add("Authorization", "secretToken")
if resp, err := client.Do(req); err != nil {
return nil, err
} else {
defer resp.Body.Close()
return readBody(resp.Body)
}
}
I finally discovered the source of the problem. It had nothing to do with the request being made, or the response being received. It had to do with the parsing of the response.
I was using bufio.NewScanner.Text to attempt to convert the response body into a string. Replacing this call with one to ioutil.ReadAll output the string that I originally expected.
Thanks for all of your help, and apologies for the misleading question.

301 status code after PostForm

I am trying to write a program which will login to ahrefs.com and parse some data.
At first I am sending GET request to ahrefs.com to get cookies and html to parse needed token:
client := &http.Client{}
jar := &myjar{}
jar.jar = make(map[string] []*http.Cookie)
client.Jar = jar
resp, _ := client.Get("https://ahrefs.com")
root, _ := html.Parse(resp.Body)
element, _ := getElementByName("_token", root)
token := ""
for _, a := range element.Attr {
if a.Key == "value" {
token = a.Val
}
}
Then I am sending POST request using PostForm to ahrefs.com/user/login/. I fill the fields with correct data (tested it via browser). When I submit form in browser it has field return_to with value of main page of the site, which should redirect to ahrefs.com/dashboard/metrics/ (the page from I want to parse data). But my program's behavior is different. After PostForm I got 301 status code:
resp, _ = client.PostForm(
"https://ahrefs.com/user/login/",
url.Values{
"email": {"djviman#gmail.com"},
"password": {"Aau4bqRxfc4ZEvu"},
"_token": {token},
"return_to": {"https://ahrefs.com/"},
})
log.Println(resp.Status)
resp.Body.Close()
Then I am sending GET request to ahrefs.com/dashboard/metrics/ but it redirects me to the home page, like I'm not logged in:
resp, _ = client.Get("https://ahrefs.com/")
log.Println(resp.Status)
resp.Body.Close()
Questions are: what I am doing wrong? And hot to successfully log in to this site?

Resources