I am taking use proxy to use POST method.
GET requests are successful, but there are problems with POST.
As I understand it, the POST request timed out, and therefore an error appears.
This is a proxy problem, or something I don't understand?
Code:
func main() {
WowProxy := getProxyFromWeb()
fmt.Println(WowProxy)
client := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(WowProxy)}}
req, err := client.PostForm("https://example.com", url.Values{"someKey" : {"SomeValue"}})
if err != nil{
panic(err.Error())
}
fmt.Println(req) }
Error:
panic: Post "https://example.com": Bad Request
Malformed request syntax, invalid request message framing, or deceptive request routing, would return a 400 Bad Request error.
400 Bad Request as per MDN
Related
I'm streaming http from Go and the server responds with "Transfer-Encoding: chunked" as expected. And I've been told that the http client in Go shall automatically dechunk the body from the http response, removing the \r\n. But in my case it isn't removed automatically, so I have to use a ChunkedReader to read the bodies.
Any idea why golang doesn't dechunk my body automatically?
EDIT:
Here is the http request:
var transport = http.Transport{
Proxy: nil,
ExpectContinueTimeout: 0,
MaxResponseHeaderBytes: 16384}
var httpClient = http.Client{
Transport: &transport,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
}}
bodyReader, bodyWriter := io.Pipe()
req, _ := http.NewRequest("GET", "http://x.x.x.x/stream", bodyReader)
response, err := httpClient.Do(req)
buffer := make([]byte, 2 << 15)
n, readErr = response.Body.Read(buffer) <-- should be dechunked body
The data read into the buffer is not dechunked. Any idea why?
I figured out why the body is not dechunked automatically. It's because the HTTP response was HTTP/1.0. In which case golang ignores the transfer encoding header.
https://github.com/golang/go/issues/12785
I am making an http post request to a server using golang. Suppose the server is currently turned off (Means the machine on which the server runs is turned off) then the request is stuck at the IP layer. So my program execution is unable to proceed further. It is unable to proceed to the Application layer. So is there any way in golang to stop this.
I am using the following code.
req, err := http.NewRequest("POST", url, bytes.NewReader(b))
if err != nil {
return errors.Wrap(err, "new request error")
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return errors.Wrap(err, "http request error")
}
defer resp.Body.Close()
Is there anything that can be added to this, to terminate the request if it doesn't find anything from the IP layer.
The default http Client has no timeout. You can create an explicit http.Client yourself and set the timeout:
var cl = &http.Client{
Timeout: time.Second * 10,
}
resp, err := cl.Do(req)
if err != nil {
// err will be set on timeout
return errors.Wrap(err, "http request error")
}
defer resp.Body.Close()
If the server does not answer any more in the middle of a request, you can handle the timeout.
Use a non-default http.Transport with its DialContext field set to a function which uses a custom context with the properly configured timeout/deadline. Another option is to use a custom net.Dialer.
Something like this:
cli := http.Client{
Transport: &http.Transport{
DialContext: func (ctx context.Context, network, address string) (net.Conn, error) {
dialer := net.Dialer{
Timeout: 3 * time.Second,
}
return dialer.DialContext(ctx, network, address)
},
},
}
req, err := http.NewRequest(...)
resp, err := cli.Do(req)
Note that as per the net.Dialer's docs the context passed to its DialContext might trump the timeout set on the dialer itself—this is
exactly what we need: the dialer's Timeout field controls exactly the
"dialing" (TCP connection establishment) while you might also arm your
HTTP request with a context (using http.Request.WithContext) controlling
the timeout of the whole request, and also be able to cancel it at any time (including the dialing step).
Playground example.
The Transport #kostix refers to is definitely what you're looking for in this case. Transports as well as Clients are safe for concurrent use as well. But please read about the Transport (and I also advise reading about the Client as well) as there are a number of different ways to affect how you handle idle connections, not just the pre-mentioned DialContext.
As you may want to set your ResponseHeaderTimeout:
ResponseHeaderTimeout, if non-zero, specifies the amount of
time to wait for a server's response headers after fully
writing the request (including its body, if any). This
time does not include the time to read the response body.
Or, if you are using a secure connection, you may want to set your TLSHandshakeTimeout:
TLSHandshakeTimeout specifies the maximum amount of time waiting to
wait for a TLS handshake. Zero means no timeout.
For readability and maintainability, I suggest also maybe creating a function to build your Client, something along the lines of:
func buildClient(timeout time.Duration) *http.Client {
tr := &http.Transport{
IdleConnTimeout: timeout,
ResponseHeaderTimeout: timeout,
TLSHandshakeTimeout: timeout,
}
client := &http.Client{
Transport: tr,
Timeout: timeout,
}
return client
}
consider this scenario!
after successful execution of a http request, what if there is an error while performing json encoding, how to override the header code
func writeResp(w http.ResponseWriter, code int, data interface{}) {
w.Header().Set("Content-Type", "application/json")
//Here I set the status to 201 StatusCreated
w.WriteHeader(code)
s := success{Data: data}
//what if there is an error here and want to override the status to 5xx error
//how to handle error here, panic?, http.Error() is not an option because as we already wrote header to 201, it just prints `http: multiple response.WriteHeader calls`
if err := json.NewEncoder(w).Encode(s); err != nil {
w.Header().Set("Content-Type", "application/json")
//it throws http: multiple response.WriteHeader calls here as we already wrote header above to 201
w.WriteHeader(code)
e := errorResponse{
Code: code,
Error: error,
Description: msg,
}
if err := json.NewEncoder(w).Encode(e); err != nil {
//same how to handle here
}
}
}
I have multiple options here, if we do just fatal logging the user won't know exactly what happened, even if I write string using w.Write([]byte(msg)) still the status says 201 created, how to respond with error code 5xx
any help is greatly appreciated
First of all, it does not seem very likely that you get an error when encoding.
See this question for reasons for Marshal to fail:
What input will cause golang's json.Marshal to return an error?
The other potential cause of error would be some problem with actually writing the data to the response stream, but in that case you'd not be able to write your custom error either.
Going back to your question, if you are concerned that encoding your object might fail, you can first Marshal your data (checking for error), then only write the 201 status code (and the encoded data) if marshalling succeeded.
Modifying your example a bit:
s := success{Data: data}
jsonData, err := json.Marshal(s)
if err != nil {
// write your error to w, then return
}
w.WriteHeader(code)
w.Header().Set("Content-Type", "application/json")
w.Write(jsonData)
Now, that last write can also throw an error.
But if that happens, it will also fail when writing your custom error, so in that case you'd better log that in the server side (or send that error to a tracker such as New Relic, etc).
I am using Go to make an HTTP GET request to an external web service. For some reason, the body of the response is always empty; the content length is always zero bytes. The response status code is always 200, however, and the call to Client.Do returns no error. The request requires an Authorization header, so I am using the http.NewRequest / http.Client.Do pattern to submit the request, as you'll see below. I have done requests similar to these in the past, but never using a GET that required a header. It seems unlikely that this the cause, but I wonder if it may be related. If anyone can spot any potential issues with the pattern used or perhaps has had a similar experience, I'd really appreciate any help.
Thank you.
if req, err := http.NewRequest("GET", "https://api.molt.in/v1/orders/11111111/items", nil); err != nil {
return nil, err
} else {
client := &http.Client{}
req.Header.Add("Authorization", "secretToken")
if resp, err := client.Do(req); err != nil {
return nil, err
} else {
defer resp.Body.Close()
return readBody(resp.Body)
}
}
I finally discovered the source of the problem. It had nothing to do with the request being made, or the response being received. It had to do with the parsing of the response.
I was using bufio.NewScanner.Text to attempt to convert the response body into a string. Replacing this call with one to ioutil.ReadAll output the string that I originally expected.
Thanks for all of your help, and apologies for the misleading question.
I would like to make an http call using an http client with these characteristics:
don't follow redirects (just get the full response of the first call, which returns a 302 redirect code)
being able to receive cookies
Hence, I am looking for a solution with http.Client rather than with trasport.RoundTrip
how can I do that?
If you use a http.Client, you have the option of using the CheckRedirect field.
This field takes a custom function which can then handle any redirects if an error is received by the initial request.
A simple example might be something like this:
client: &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return errors.New("something bad happened") // or maybe the error from the request
},
}
You could also return the http.ErrUseLastResponse error which returns the most recent response with its body unclosed.
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
According to https://pkg.go.dev/net/http#Client.CheckRedirect