Terminate http request from IP layer using golang - http

I am making an http post request to a server using golang. Suppose the server is currently turned off (Means the machine on which the server runs is turned off) then the request is stuck at the IP layer. So my program execution is unable to proceed further. It is unable to proceed to the Application layer. So is there any way in golang to stop this.
I am using the following code.
req, err := http.NewRequest("POST", url, bytes.NewReader(b))
if err != nil {
return errors.Wrap(err, "new request error")
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return errors.Wrap(err, "http request error")
}
defer resp.Body.Close()
Is there anything that can be added to this, to terminate the request if it doesn't find anything from the IP layer.

The default http Client has no timeout. You can create an explicit http.Client yourself and set the timeout:
var cl = &http.Client{
Timeout: time.Second * 10,
}
resp, err := cl.Do(req)
if err != nil {
// err will be set on timeout
return errors.Wrap(err, "http request error")
}
defer resp.Body.Close()
If the server does not answer any more in the middle of a request, you can handle the timeout.

Use a non-default http.Transport with its DialContext field set to a function which uses a custom context with the properly configured timeout/deadline. Another option is to use a custom net.Dialer.
Something like this:
cli := http.Client{
Transport: &http.Transport{
DialContext: func (ctx context.Context, network, address string) (net.Conn, error) {
dialer := net.Dialer{
Timeout: 3 * time.Second,
}
return dialer.DialContext(ctx, network, address)
},
},
}
req, err := http.NewRequest(...)
resp, err := cli.Do(req)
Note that as per the net.Dialer's docs the context passed to its DialContext might trump the timeout set on the dialer itself—this is
exactly what we need: the dialer's Timeout field controls exactly the
"dialing" (TCP connection establishment) while you might also arm your
HTTP request with a context (using http.Request.WithContext) controlling
the timeout of the whole request, and also be able to cancel it at any time (including the dialing step).
Playground example.

The Transport #kostix refers to is definitely what you're looking for in this case. Transports as well as Clients are safe for concurrent use as well. But please read about the Transport (and I also advise reading about the Client as well) as there are a number of different ways to affect how you handle idle connections, not just the pre-mentioned DialContext.
As you may want to set your ResponseHeaderTimeout:
ResponseHeaderTimeout, if non-zero, specifies the amount of
time to wait for a server's response headers after fully
writing the request (including its body, if any). This
time does not include the time to read the response body.
Or, if you are using a secure connection, you may want to set your TLSHandshakeTimeout:
TLSHandshakeTimeout specifies the maximum amount of time waiting to
wait for a TLS handshake. Zero means no timeout.
For readability and maintainability, I suggest also maybe creating a function to build your Client, something along the lines of:
func buildClient(timeout time.Duration) *http.Client {
tr := &http.Transport{
IdleConnTimeout: timeout,
ResponseHeaderTimeout: timeout,
TLSHandshakeTimeout: timeout,
}
client := &http.Client{
Transport: tr,
Timeout: timeout,
}
return client
}

Related

Golang HTTP Get Request Not Resolving for some URL

I was trying to build some sort of website status checker. I figure out that the golang HTTP get request is not resolved and hung forever for some URL like https://www.hetzner.com. But the same URL works if we do curl.
Golang
Here there is no error thrown. It just hangs on http.Get
func main() {
resp, err := http.Get("https://www.hetzner.com")
if err != nil {
fmt.Println("Error while retrieving site", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("Eroor while reading response body", err)
}
fmt.Println("RESPONSE", string(body))}
CURL
I get the response while running following command.
curl https://www.hetzner.com
What may be the reason? And how do I resolve this issue from golang HTTP?
Your specific case can be fixed by specifying HTTP User-Agent Header:
import (
"fmt"
"io"
"net/http"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://www.hetzner.com", nil)
if err != nil {
fmt.Println("Error while retrieving site", err)
}
req.Header.Set("User-Agent", "Golang_Spider_Bot/3.0")
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error while retrieving site", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("Eroor while reading response body", err)
}
fmt.Println("RESPONSE", string(body))
}
Note: many other hosts will reject requests from your server because of some security rules on their side. Some ideas:
Empty or bot-like User-Agent HTTP header
Location of your IP address. For example, online shops in the USA don't need to handle requests from Russia.
Autonomous System or CIDR of your provider. Some ASNs are completely blackholed because of the enormous malicious activities from their residents.
Note 2: Many modern websites have DDoS protection or CDN systems in front of them. If Cloudflare protects your target website, your HTTP request will be blocked despite the status code 200. To handle this, you need to build something able to render JavaScript-based websites and add some scripts to resolve a captcha.
Also, if you check a considerable amount of websites in a short time, you will be blocked by your DNS servers as they have some inbuild rate limits. In this case, you may want to take a look at massdns or similar solutions.

Go HTTP RoundTripper: Preventing Connection Reuse Based on Response

I have a use case where I want to use an HTTP client in Go with pooled connections (connection re-use), but with the special case where a connection is intentionally closed (not allowed for re-use) if a request on that connection returns a specific HTTP status code.
I've implemented a custom http.RoundTripper, which wraps an http.Transport, and can inspect the response status code. However, I can't seem to find a way to prevent the http.Transport from re-using that connection, without also preventing it from re-using any other connection.
Is this possible using the net/http package? If not, any suggested workaround for accomplishing this?
My current code looks something like this:
type MyTransport struct {
transport *http.Transport
}
func (mt *MyTransport) RoundTrip(req *http.Request) (*http.Response, error) {
resp, err := tt.transport.RoundTrip(req)
if err != nil {
return resp, err
}
if resp.StatusCode == 567 {
// HERE:
// Do something to prevent re-use of this connection
}
return resp, err
}

How to know proxy used by http client in given request

I'm doing some requests through some proxy servers. The function that defines which proxy url to use will choose randomly from a list of proxies. I would like to know for a given request, which proxy url is being used. As far as I know, when using a proxy server the http headers remain the same, but the tcp headers are the one that change.
Here's some code illustrating it (no error handling for simplicity):
func main() {
transport := &http.Transport{Proxy: chooseProxy}
client := http.Client{Transport: transport}
request, err := http.NewRequest(http.MethodGet, "https://www.google.com", nil)
checkErr(err)
// How to know here which proxy was used? Suppose the same client will perform several requests to different URL's.
response, err := client.Do(request)
checkErr(err)
dump, _ := httputil.DumpRequest(response.Request, false)
fmt.Println(dump)
}
func chooseProxy(request *http.Request) (*url.URL, error) {
proxies := []string{"proxy1", "proxy2", "proxy3"}
proxyToUse := proxies[rand.Intn(len(proxies))]
return url.Parse(proxyToUse)
}
I'm assuming that the Proxy function in the transport is called for each request even if the same client is used, as per the docs that say "Proxy specifies a function to return a proxy for a given Request". Am I right?
Some HTTP proxies add a Via header that tell who they are.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Via
You can modify your chooseProxy function so that it saves the proxy selected.
To do that, you can transform the chooseProxy func into a method of a type that will be used as storage for the information you want to keep:
type proxySelector string
func (sel *proxySelector) chooseProxy(request *http.Request) (*url.URL, error) {
proxies := []string{"proxy1", "proxy2", "proxy3"}
proxyToUse := proxies[rand.Intn(len(proxies))]
*sel = proxySelector(proxyToUse) // <-----
return url.Parse(proxyToUse)
}
func main() {
var proxy proxySelector
transport := &http.Transport{Proxy: proxy.chooseProxy} // <-----
client := http.Client{Transport: transport}
request, err := http.NewRequest(http.MethodGet, "https://www.google.com", nil)
checkErr(err)
// How to know here which proxy was used? Suppose the same client will perform several requests to different URL's.
response, err := client.Do(request)
checkErr(err)
dump, _ := httputil.DumpRequest(response.Request, false)
fmt.Println(dump)
fmt.Println("Proxy:", string(proxy)) // <-----
}
The request which contains the target URI is given as argument request to chooseProxy. So you can have the correct mapping already inside your chooseProxy function, all you need to to is check proxyToUse vs. request.URL there.
If you don't really trust the code that this mapping is actually done, then you need to look outside the code. For example you can look at the actual network traffic with Wireshark to see which proxy gets accessed.

Why is there a 60 second delay on my HTTP POST request when using a Go HTTP client?

My goal is to scrape a website that requires me to log in first using HTTP requests in Golang. I actually succeeded by finding out I can send a post request to the website writing form-data into the body of the request. When I test this through an API development software I use called Postman, the response is instantaneous with no delays. However, when performing the request with an HTTP client in Go, there is a consistent 60 second delay every single time. I end up getting a logged in page, but for my program I need the response to be nearly instantaneous.
As you can see in my code, I've tried adding a bunch of headers to the request like "Connection", "Content-Type", "User-Agent" since I thought maaaaaybe the website can tell I'm requesting from a program and is forcing me to wait 60 seconds for a response. Adding these headers to make my request more legitimate(?) doesn't work at all.
Is the delay coming from Go's HTTP client being slow or is there something wrong with how I'm forming my HTTP POST request? Also, was I on to something with my headers and HTTP client is rewriting them when they send out?
Here's my simple program...
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
"net/http/cookiejar"
"os"
)
func main() {
url := "https://easypronunciation.com/en/log-in"
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
_ = writer.WriteField("email", "foo#bar.com")
_ = writer.WriteField("password", "*********")
_ = writer.WriteField("persistent_login", "on")
_ = writer.WriteField("submit", "")
err := writer.Close()
if err != nil {
fmt.Println(err)
}
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", writer.FormDataContentType())
req.Header.Set("Connection", "Keep-Alive")
req.Header.Set("Accept-Language", "en-US")
req.Header.Set("User-Agent", "Mozilla/5.0")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
f, err := os.Create("response.html")
defer f.Close()
res.Write(f)
}
I doubt, this is the go client library too. I would suggest printing out the latencies for different components and see if/where the 60 second delay is. I would also replace and try different URLs instead

Sending HTTP Put body from ReadCloser never ends

Target
I want to send to data to my server from a read closer. (In the example a NopCloser, later it will be the Stdout of an exec.Command)
Problem
The Request never ends. Even if I manually close the cmdOut the program nevers ends. Concrete: It never reaches the "Request Done" line and there by never calls wg.Done()
Gotchas
All the data is sent correctly to the server (even with the exec.Command Stdout). But the http.DefaultClient.Do seems to be still listening on the ReadCloser after it is empty (and closed in the main routine)
Code
cmdOut := ioutil.NopCloser(bytes.NewBuffer([]byte("Hallo DU")))
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
// localhost:1234 is a netcat server: "nc -l -p 1234"
req, _ := http.NewRequest("PUT", "http://localhost:1234", cmdOut)
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Println(err)
return
}
// Never reaches this line
log.Println("Request Done")
}()
cmdOut.Close()
log.Println("Wait for go routine")
wg.Wait()
log.Println("DONE")
The problem isn't the sending of your request--it's the receiving of the response.
You're sending your request to netcat, which simply discards the request, and does nothing else. This leaves the HTTP client library waiting for an HTTP response, which never comes.
The solution is to talk to an actual HTTP server.

Resources