I am using Go to make an HTTP GET request to an external web service. For some reason, the body of the response is always empty; the content length is always zero bytes. The response status code is always 200, however, and the call to Client.Do returns no error. The request requires an Authorization header, so I am using the http.NewRequest / http.Client.Do pattern to submit the request, as you'll see below. I have done requests similar to these in the past, but never using a GET that required a header. It seems unlikely that this the cause, but I wonder if it may be related. If anyone can spot any potential issues with the pattern used or perhaps has had a similar experience, I'd really appreciate any help.
Thank you.
if req, err := http.NewRequest("GET", "https://api.molt.in/v1/orders/11111111/items", nil); err != nil {
return nil, err
} else {
client := &http.Client{}
req.Header.Add("Authorization", "secretToken")
if resp, err := client.Do(req); err != nil {
return nil, err
} else {
defer resp.Body.Close()
return readBody(resp.Body)
}
}
I finally discovered the source of the problem. It had nothing to do with the request being made, or the response being received. It had to do with the parsing of the response.
I was using bufio.NewScanner.Text to attempt to convert the response body into a string. Replacing this call with one to ioutil.ReadAll output the string that I originally expected.
Thanks for all of your help, and apologies for the misleading question.
Related
I was trying to build some sort of website status checker. I figure out that the golang HTTP get request is not resolved and hung forever for some URL like https://www.hetzner.com. But the same URL works if we do curl.
Golang
Here there is no error thrown. It just hangs on http.Get
func main() {
resp, err := http.Get("https://www.hetzner.com")
if err != nil {
fmt.Println("Error while retrieving site", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("Eroor while reading response body", err)
}
fmt.Println("RESPONSE", string(body))}
CURL
I get the response while running following command.
curl https://www.hetzner.com
What may be the reason? And how do I resolve this issue from golang HTTP?
Your specific case can be fixed by specifying HTTP User-Agent Header:
import (
"fmt"
"io"
"net/http"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://www.hetzner.com", nil)
if err != nil {
fmt.Println("Error while retrieving site", err)
}
req.Header.Set("User-Agent", "Golang_Spider_Bot/3.0")
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error while retrieving site", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("Eroor while reading response body", err)
}
fmt.Println("RESPONSE", string(body))
}
Note: many other hosts will reject requests from your server because of some security rules on their side. Some ideas:
Empty or bot-like User-Agent HTTP header
Location of your IP address. For example, online shops in the USA don't need to handle requests from Russia.
Autonomous System or CIDR of your provider. Some ASNs are completely blackholed because of the enormous malicious activities from their residents.
Note 2: Many modern websites have DDoS protection or CDN systems in front of them. If Cloudflare protects your target website, your HTTP request will be blocked despite the status code 200. To handle this, you need to build something able to render JavaScript-based websites and add some scripts to resolve a captcha.
Also, if you check a considerable amount of websites in a short time, you will be blocked by your DNS servers as they have some inbuild rate limits. In this case, you may want to take a look at massdns or similar solutions.
I have a use case where I want to use an HTTP client in Go with pooled connections (connection re-use), but with the special case where a connection is intentionally closed (not allowed for re-use) if a request on that connection returns a specific HTTP status code.
I've implemented a custom http.RoundTripper, which wraps an http.Transport, and can inspect the response status code. However, I can't seem to find a way to prevent the http.Transport from re-using that connection, without also preventing it from re-using any other connection.
Is this possible using the net/http package? If not, any suggested workaround for accomplishing this?
My current code looks something like this:
type MyTransport struct {
transport *http.Transport
}
func (mt *MyTransport) RoundTrip(req *http.Request) (*http.Response, error) {
resp, err := tt.transport.RoundTrip(req)
if err != nil {
return resp, err
}
if resp.StatusCode == 567 {
// HERE:
// Do something to prevent re-use of this connection
}
return resp, err
}
My goal is to scrape a website that requires me to log in first using HTTP requests in Golang. I actually succeeded by finding out I can send a post request to the website writing form-data into the body of the request. When I test this through an API development software I use called Postman, the response is instantaneous with no delays. However, when performing the request with an HTTP client in Go, there is a consistent 60 second delay every single time. I end up getting a logged in page, but for my program I need the response to be nearly instantaneous.
As you can see in my code, I've tried adding a bunch of headers to the request like "Connection", "Content-Type", "User-Agent" since I thought maaaaaybe the website can tell I'm requesting from a program and is forcing me to wait 60 seconds for a response. Adding these headers to make my request more legitimate(?) doesn't work at all.
Is the delay coming from Go's HTTP client being slow or is there something wrong with how I'm forming my HTTP POST request? Also, was I on to something with my headers and HTTP client is rewriting them when they send out?
Here's my simple program...
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
"net/http/cookiejar"
"os"
)
func main() {
url := "https://easypronunciation.com/en/log-in"
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
_ = writer.WriteField("email", "foo#bar.com")
_ = writer.WriteField("password", "*********")
_ = writer.WriteField("persistent_login", "on")
_ = writer.WriteField("submit", "")
err := writer.Close()
if err != nil {
fmt.Println(err)
}
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", writer.FormDataContentType())
req.Header.Set("Connection", "Keep-Alive")
req.Header.Set("Accept-Language", "en-US")
req.Header.Set("User-Agent", "Mozilla/5.0")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
f, err := os.Create("response.html")
defer f.Close()
res.Write(f)
}
I doubt, this is the go client library too. I would suggest printing out the latencies for different components and see if/where the 60 second delay is. I would also replace and try different URLs instead
Below handler handles GET request, without populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Below handler handles GET request, populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
rw.Header().Add("Content-Type", "application/json")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Both cases are working fine with simple curl request.
For any http client,
When do we need to populate content-type header, before sending the response, to client?
Always read the documentation first!
The answer to this is clearly covered here (emphasis obviously added):
// If WriteHeader has not yet been called, Write calls
// WriteHeader(http.StatusOK) before writing the data. If the Header
// does not contain a Content-Type line, Write adds a Content-Type set
// to the result of passing the initial 512 bytes of written data to
// DetectContentType. Additionally, if the total size of all written
// data is under a few KB and there are no Flush calls, the
// Content-Length header is added automatically.
To explicitly answer your secondary question:
When do we need to populate content-type header?
Any time you don't want it to be automatically detected. Automatic detection is imprecise, so you generally don't want to rely on it.
consider this scenario!
after successful execution of a http request, what if there is an error while performing json encoding, how to override the header code
func writeResp(w http.ResponseWriter, code int, data interface{}) {
w.Header().Set("Content-Type", "application/json")
//Here I set the status to 201 StatusCreated
w.WriteHeader(code)
s := success{Data: data}
//what if there is an error here and want to override the status to 5xx error
//how to handle error here, panic?, http.Error() is not an option because as we already wrote header to 201, it just prints `http: multiple response.WriteHeader calls`
if err := json.NewEncoder(w).Encode(s); err != nil {
w.Header().Set("Content-Type", "application/json")
//it throws http: multiple response.WriteHeader calls here as we already wrote header above to 201
w.WriteHeader(code)
e := errorResponse{
Code: code,
Error: error,
Description: msg,
}
if err := json.NewEncoder(w).Encode(e); err != nil {
//same how to handle here
}
}
}
I have multiple options here, if we do just fatal logging the user won't know exactly what happened, even if I write string using w.Write([]byte(msg)) still the status says 201 created, how to respond with error code 5xx
any help is greatly appreciated
First of all, it does not seem very likely that you get an error when encoding.
See this question for reasons for Marshal to fail:
What input will cause golang's json.Marshal to return an error?
The other potential cause of error would be some problem with actually writing the data to the response stream, but in that case you'd not be able to write your custom error either.
Going back to your question, if you are concerned that encoding your object might fail, you can first Marshal your data (checking for error), then only write the 201 status code (and the encoded data) if marshalling succeeded.
Modifying your example a bit:
s := success{Data: data}
jsonData, err := json.Marshal(s)
if err != nil {
// write your error to w, then return
}
w.WriteHeader(code)
w.Header().Set("Content-Type", "application/json")
w.Write(jsonData)
Now, that last write can also throw an error.
But if that happens, it will also fail when writing your custom error, so in that case you'd better log that in the server side (or send that error to a tracker such as New Relic, etc).