I am trying to write a program which will login to ahrefs.com and parse some data.
At first I am sending GET request to ahrefs.com to get cookies and html to parse needed token:
client := &http.Client{}
jar := &myjar{}
jar.jar = make(map[string] []*http.Cookie)
client.Jar = jar
resp, _ := client.Get("https://ahrefs.com")
root, _ := html.Parse(resp.Body)
element, _ := getElementByName("_token", root)
token := ""
for _, a := range element.Attr {
if a.Key == "value" {
token = a.Val
}
}
Then I am sending POST request using PostForm to ahrefs.com/user/login/. I fill the fields with correct data (tested it via browser). When I submit form in browser it has field return_to with value of main page of the site, which should redirect to ahrefs.com/dashboard/metrics/ (the page from I want to parse data). But my program's behavior is different. After PostForm I got 301 status code:
resp, _ = client.PostForm(
"https://ahrefs.com/user/login/",
url.Values{
"email": {"djviman#gmail.com"},
"password": {"Aau4bqRxfc4ZEvu"},
"_token": {token},
"return_to": {"https://ahrefs.com/"},
})
log.Println(resp.Status)
resp.Body.Close()
Then I am sending GET request to ahrefs.com/dashboard/metrics/ but it redirects me to the home page, like I'm not logged in:
resp, _ = client.Get("https://ahrefs.com/")
log.Println(resp.Status)
resp.Body.Close()
Questions are: what I am doing wrong? And hot to successfully log in to this site?
Related
I am trying to retrieve the filename from this http writer for testing purposes.
On the server I have:
func servefile(w http.ResponseWriter, r *http.Request ) {
...
// file.Name() is randomized with os.CreateTemp(dir, temp+"*"+ext) above
w.Header().Set("Content-Disposition", "attachment; filename="+file.Name())
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeFile(w, r, file.Name()+".xyz") // serve file to user to download
...
}
*I put .xyz as a place holder for this demonstration
I am testing this function programatically with Go and I want to access the filename to able to save it in a variable in the client code.
I have looked at this post How can I parse the Content-Disposition header to retrieve the filename property? , but I have not gotten it to work. I have no clue what the filename is on the client side so I don't know how to reference it specifically. I know my code on the server side works, because when I send a request (through the browser) to this endpoint/function, the "Downloads" popup shows the file download progress with the name of the file.
EDIT** This is the client code I am calling it from:
func TestGetFile(t *testing.T) {
...
cid := "some string"
// requestfile() creates, executes, and returns an httptest.ResponseRecorder to the requestFile endpoint
reqfileRespRecorder := requestfile()
// createTmpFile creates a new file out of the contents recieved in requestfile()
filePath := "/tmp/temp.xyz"
file := createTmpFile(reqfileRespRecorder , filePath)
// CreateWriter() - writes file contents to body of multipart.Writer
w, body := createWriter(file)
// Create request to postRecord endpoint
req, err := http.NewRequest("POST", "/PostRecord?CID="+cid, body)
check(err)
req.Header.Add("Content-Type", w.FormDataContentType())
// execute request to PostRecord endpoint. returns an httptest.ResponseRecorder
respRecorder := executeRequest(PostRecord, req)
disposition, params, err := mime.ParseMediaType(`Content-Disposition`)
...
}
Based on #BaytaDarrell's comment. It dawned on me that I could print out the responses. That helped me realize that I was trying to find the content-disposition after the wrong request/response. The linked post still didn't help, but I got my code working like this:
func TestGetFile(t *testing.T) {
...
cid := "some string"
// requestfile() creates, executes, and returns an httptest.ResponseRecorder to the requestFile endpoint
reqfileRespRecorder := requestfile()
disp := reqfileRespRecorder.Header().Get("Content-Disposition")
line := strings.Split(disp, "=")
filename := line[1]
fmt.Println("filename: ", filename)
// createTmpFile creates a new file out of the contents recieved in requestfile()
filePath := "/tmp/temp.xyz"
file := createTmpFile(reqfileRespRecorder , filePath)
// CreateWriter() - writes file contents to body of multipart.Writer
w, body := createWriter(file)
...
}
Their comment realized I should re-look at the httptest package documentation. Here I found the Header() function and that I can use it to look at the header with Get().
This line reqfileRespRecorder.Header().Get("Content-Disposition") returns attachment; filename=temp37bf73gd.xyz and to store the filename in a variable I split on =.
I'm trying to extract a value from an HTTP POST request body (in my simple Go HTTP server) using the net/http PostFormValue and my output is an empty string when I'm looking for the any key in general, but in my case trying to fetch the hub.secret for use in a HMAC check. I use Postman to send the request to my localhost:8080 instance using the Gorilla/mux router, with header Content-Type: application/x-www-form-urlencoded set.
My handler looks like so:
func rootPostHandler(w http.ResponseWriter, r *http.Request) {
var expectedMac []byte
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("r.Body is:", string(body)) // debug: print the request POST body
message := body // debug: set message just for extra clarity
errParse := r.ParseForm()
if errParse != nil {
// handle err
}
secret := []byte(r.PostFormValue("hub.secret"))
log.Println("secret is: ", string(secret))
mac := hmac.New(sha256.New, secret)
mac.Write(message)
expectedMac = mac.Sum(nil)
fmt.Println("Is HMAC equal? ", hmac.Equal(message, expectedMac))
w.Header().Add("X-Hub-Signature", "sha256="+string(message))
}
The r.Body:
hub.callback=http%253A%252F%252Fweb-sub-client%253A8080%252FbRxvcmOcNk&hub.mode=subscribe&hub.secret=xTgSGLOtPNrBLLgYcKnL&hub.topic=%252Fa%252Ftopic
And the output for print secret etc is empty string, meaning it can't find hub.secret, right? What am I missing here?
The application reads the request body to EOF on this line:
body, err := ioutil.ReadAll(r.Body)
ParseForm returns an empty form because the body is at EOF at this line:
errParse := r.ParseForm()
The request body is read from the network connection. The request body cannot be read a second time.
Remove the call to ioutil.ReadAll or create a new body reader using the data returned from ioutil.ReadAll:
r.Body = io.NopCloser(bytes.NewReader(body))
My goal is to scrape a website that requires me to log in first using HTTP requests in Golang. I actually succeeded by finding out I can send a post request to the website writing form-data into the body of the request. When I test this through an API development software I use called Postman, the response is instantaneous with no delays. However, when performing the request with an HTTP client in Go, there is a consistent 60 second delay every single time. I end up getting a logged in page, but for my program I need the response to be nearly instantaneous.
As you can see in my code, I've tried adding a bunch of headers to the request like "Connection", "Content-Type", "User-Agent" since I thought maaaaaybe the website can tell I'm requesting from a program and is forcing me to wait 60 seconds for a response. Adding these headers to make my request more legitimate(?) doesn't work at all.
Is the delay coming from Go's HTTP client being slow or is there something wrong with how I'm forming my HTTP POST request? Also, was I on to something with my headers and HTTP client is rewriting them when they send out?
Here's my simple program...
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
"net/http/cookiejar"
"os"
)
func main() {
url := "https://easypronunciation.com/en/log-in"
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
_ = writer.WriteField("email", "foo#bar.com")
_ = writer.WriteField("password", "*********")
_ = writer.WriteField("persistent_login", "on")
_ = writer.WriteField("submit", "")
err := writer.Close()
if err != nil {
fmt.Println(err)
}
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", writer.FormDataContentType())
req.Header.Set("Connection", "Keep-Alive")
req.Header.Set("Accept-Language", "en-US")
req.Header.Set("User-Agent", "Mozilla/5.0")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
f, err := os.Create("response.html")
defer f.Close()
res.Write(f)
}
I doubt, this is the go client library too. I would suggest printing out the latencies for different components and see if/where the 60 second delay is. I would also replace and try different URLs instead
So I'm writing a basic webapp and I'm having trouble redirecting the user after a sucessfull login. The code is:
t, err := template.ParseFiles("home.html")
if err != nil {
log.Fatal("template.ParseFiles: ", err)
}
err = t.Execute(w, nil)
if err != nil {
log.Fatal("t.Execute: ", err)
}
if r.Method == "POST" {
r.ParseForm()
user := r.FormValue("username")
pass := r.FormValue("password")
if checkLogin(user, pass) {
loggedIn = true
http.Redirect(w, r, "/home", 302)
}
}
The error message is: "http: multiple response.WriteHeader calls".
My problem is that I don't see a way to serve the html file containing the login-form without calling t.Execute which sets the header.
How can I display the login page and still be able to redirect to a different page?
You are writing (using w) and then later trying to redirect (also using w) using 302 header redirection.
You can only send headers once, and if you start writing to w it assumes a 200 header (OK)
Also, Its best if you check the http.Method before writing to the ResponseWriter (w)
And, Remember to return after a redirection or handing over the ResponseWriter and Request pair to another function!
Hope this helps.
How can I display the login page and still be able to redirect to a different page?
Have a different route for authentication. Make the login form submit to the authentication route. Have a separate handler for authentication as well.
For example, your login form:
<form method="post" action="/auth">
your Go main:
http.HandleFunc("/", homeHandler)
http.HandleFunc("/auth", authHandler)
When authentication processing is complete you can redirect the user to the appropriate page. You could pass a parameter in the query string that contains the destination path for the redirect.
I'm trying to send a GET request to the following api:
https://poloniex.com/public?command=returnOrderBook
w/ URL parameters:
currencyPair=BTC_ETH
depth=20
--> ¤cyPair=BTC_ETH&depth=20
I try to setup and execute my request as so: (note I've removed error checking for brevity)
pair := "BTC_ETH"
depth := 20
reqURL := "https://poloniex.com/public?command=returnOrderBook"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
fmt.Printf("\n Values = %s\n", values.Encode()) //DEBUG
req, err := http.NewRequest("GET", reqURL, strings.NewReader(values.Encode()))
fmt.Printf("\nREQUEST = %+v\n", req) //DEBUG
resp, err := api.client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Printf("\nREST CALL RETURNED: %X\n",body) //DEBUG
My DEBUG print statements print out the following:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:{Reader:0xc82028e840} ContentLength:29 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: {"error":"Please specify a currency pair."}
Playing around with Postman I figured out the API only returns this error when the currencyPair parameter is not specified (including miscapitalized). I can't figure out why the request doesn't include the URL parameters I specified as it's obvious from my debug print statements that the values.Encode() is correct. The content length in the request corresponds to the right amount of chars (bytes) needed for URL parameters.
Now after playing around a bit I found a solution.
If I replace the http.NewRequest() line with the following it works:
req, err := http.NewRequest(HTTPType, reqURL + "&" + values.Encode(), nil)
However, it's really bothering me why the original statement doesn't work.
The new DEBUG output is:
Values = currencyPair=BTC_ETH&depth=20
REQUEST = &{Method:GET URL:https://poloniex.com/public?command=returnOrderBook¤cyPair=BTC_ETH&depth=5 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[Poloniex GO API Agent]] Body:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:poloniex.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil>}
REST CALL RETURNED: *way too long, just assume it's the correct financial data*
Would love some input on what I did wrong in the original statement. I used the same method (original) for a different api endpoint w/ URL parameters and it worked fine. Confused on why it didn't work in this case.
GET requests should not contain a body. Instead, you need to put the form into the query string.
Here's the proper way to do that, without hacky string concatenation:
reqURL := "https://poloniex.com/public"
values := url.Values { "currencyPair": []string{pair}, "depth": []string{depth}}
values.Set("command", "returnOrderBook")
uri, _ := url.Parse(reqURL)
uri.Query = values.Encode()
reqURL = uri.String()
fmt.Println(reqURL)
req, err := http.NewRequest("GET", reqURL, nil)
if err != nil {
panic(err) // NewRequest only errors on bad methods or un-parsable urls
}
https://play.golang.org/p/ZCLUu7UgZL