I'm streaming http from Go and the server responds with "Transfer-Encoding: chunked" as expected. And I've been told that the http client in Go shall automatically dechunk the body from the http response, removing the \r\n. But in my case it isn't removed automatically, so I have to use a ChunkedReader to read the bodies.
Any idea why golang doesn't dechunk my body automatically?
EDIT:
Here is the http request:
var transport = http.Transport{
Proxy: nil,
ExpectContinueTimeout: 0,
MaxResponseHeaderBytes: 16384}
var httpClient = http.Client{
Transport: &transport,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
}}
bodyReader, bodyWriter := io.Pipe()
req, _ := http.NewRequest("GET", "http://x.x.x.x/stream", bodyReader)
response, err := httpClient.Do(req)
buffer := make([]byte, 2 << 15)
n, readErr = response.Body.Read(buffer) <-- should be dechunked body
The data read into the buffer is not dechunked. Any idea why?
I figured out why the body is not dechunked automatically. It's because the HTTP response was HTTP/1.0. In which case golang ignores the transfer encoding header.
https://github.com/golang/go/issues/12785
Related
I'm developing a Security Scanner and therefore need to send HTTP requests which don't honor RFC specifications. However, golang is very strict to comply with these.
Issue
I want to send a HTTP request which contains prohibited special characters such as "".
For example: "Ill\egal": "header value"
However, golang always throws the error: 'net/http: invalid header field name "Ill\egal"'.
This error is thrown on line 523 at https://go.dev/src/net/http/transport.go
Issue
I want to send a single HTTP request which contains either two content-length, two transfer-encoding or one content-length & one transfer-encoding header (for HTTP request smuggling). Those need sometimes to have wrong values.
However, it isn't possible to set those headers oneself, they are generated automatically. So it's only possible to use one of these headers with a correct value.
I've bypassed this by using a Raw TCP Stream, however this solution isn't satisfying, as I can't use a proxy this way: Use Dialer with Proxy. Route TCP stream through Proxy
Issue
I want to send a HTTP request where the header name is mixed upper and lowercase. E.g. "HeAdErNaMe": "header value".
This is possible for HTTP 1 requests by writing directly to the header map (req.Header["HeAdErNaMe"] = []string{"header value"})
However for HTTP 2 requests the headers will still be capitalized to meet the RFC specifications.
You can dump request into a buffer, modify the buffer (with regexp or replace), and send modified buffer to the host using net.Dial.
Example:
package main
import (
"bufio"
"crypto/tls"
"fmt"
"log"
"net/http"
"net/http/httputil"
"strings"
)
func main() {
// create and dump request
req, err := http.NewRequest(http.MethodGet, "https://golang.org", nil)
if err != nil {
log.Fatal(err)
}
req.Header.Add("User-Agent", "aaaaa")
buf, err := httputil.DumpRequest(req, true)
if err != nil {
log.Fatal(err)
}
// Corrupt request
str := string(buf)
str = strings.Replace(str, "User-Agent: aaaaa", "UsEr-AgEnT: aaa\"aaa", 1)
println(str)
// Dial and send raw request text
conn, err := tls.Dial("tcp", "golang.org:443", nil)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
fmt.Fprintf(conn, str)
// Read response
br := bufio.NewReader(conn)
resp, err := http.ReadResponse(br, nil)
if err != nil {
log.Fatal(err)
}
log.Printf("%+v", resp)
}
I'm trying to extract a value from an HTTP POST request body (in my simple Go HTTP server) using the net/http PostFormValue and my output is an empty string when I'm looking for the any key in general, but in my case trying to fetch the hub.secret for use in a HMAC check. I use Postman to send the request to my localhost:8080 instance using the Gorilla/mux router, with header Content-Type: application/x-www-form-urlencoded set.
My handler looks like so:
func rootPostHandler(w http.ResponseWriter, r *http.Request) {
var expectedMac []byte
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("r.Body is:", string(body)) // debug: print the request POST body
message := body // debug: set message just for extra clarity
errParse := r.ParseForm()
if errParse != nil {
// handle err
}
secret := []byte(r.PostFormValue("hub.secret"))
log.Println("secret is: ", string(secret))
mac := hmac.New(sha256.New, secret)
mac.Write(message)
expectedMac = mac.Sum(nil)
fmt.Println("Is HMAC equal? ", hmac.Equal(message, expectedMac))
w.Header().Add("X-Hub-Signature", "sha256="+string(message))
}
The r.Body:
hub.callback=http%253A%252F%252Fweb-sub-client%253A8080%252FbRxvcmOcNk&hub.mode=subscribe&hub.secret=xTgSGLOtPNrBLLgYcKnL&hub.topic=%252Fa%252Ftopic
And the output for print secret etc is empty string, meaning it can't find hub.secret, right? What am I missing here?
The application reads the request body to EOF on this line:
body, err := ioutil.ReadAll(r.Body)
ParseForm returns an empty form because the body is at EOF at this line:
errParse := r.ParseForm()
The request body is read from the network connection. The request body cannot be read a second time.
Remove the call to ioutil.ReadAll or create a new body reader using the data returned from ioutil.ReadAll:
r.Body = io.NopCloser(bytes.NewReader(body))
My goal is to scrape a website that requires me to log in first using HTTP requests in Golang. I actually succeeded by finding out I can send a post request to the website writing form-data into the body of the request. When I test this through an API development software I use called Postman, the response is instantaneous with no delays. However, when performing the request with an HTTP client in Go, there is a consistent 60 second delay every single time. I end up getting a logged in page, but for my program I need the response to be nearly instantaneous.
As you can see in my code, I've tried adding a bunch of headers to the request like "Connection", "Content-Type", "User-Agent" since I thought maaaaaybe the website can tell I'm requesting from a program and is forcing me to wait 60 seconds for a response. Adding these headers to make my request more legitimate(?) doesn't work at all.
Is the delay coming from Go's HTTP client being slow or is there something wrong with how I'm forming my HTTP POST request? Also, was I on to something with my headers and HTTP client is rewriting them when they send out?
Here's my simple program...
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
"net/http/cookiejar"
"os"
)
func main() {
url := "https://easypronunciation.com/en/log-in"
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
_ = writer.WriteField("email", "foo#bar.com")
_ = writer.WriteField("password", "*********")
_ = writer.WriteField("persistent_login", "on")
_ = writer.WriteField("submit", "")
err := writer.Close()
if err != nil {
fmt.Println(err)
}
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", writer.FormDataContentType())
req.Header.Set("Connection", "Keep-Alive")
req.Header.Set("Accept-Language", "en-US")
req.Header.Set("User-Agent", "Mozilla/5.0")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
f, err := os.Create("response.html")
defer f.Close()
res.Write(f)
}
I doubt, this is the go client library too. I would suggest printing out the latencies for different components and see if/where the 60 second delay is. I would also replace and try different URLs instead
i want get a Request , and use ParseForm to check some data , then send the same one to proxy , buy if i do that , we have a problem
error log like this
2020/05/26 15:34:47 http: proxy error: net/http: HTTP/1.x transport connection broken: http: ContentLength=32 with Body length 0
so I finally figured it out , ParseForm () well close Request .body
and this is work proxy code
url, _ := url.Parse(config.GetGameHost())
proxy := httputil.NewSingleHostReverseProxy(url)
r.URL.Host = url.Host
r.URL.Scheme = url.Scheme
r.Header.Set("X-Forwarded-Host", r.Header.Get("Host"))
r.Host = url.Host
proxy.ServeHTTP(w, r)
so , i think i need clone the deep clone request too get my data and send origin request too proxy
i edit my code too this
nr := r.Clone(r.Context())
nr.ParseForm()
url, _ := url.Parse(config.GetGameHost())
proxy := httputil.NewSingleHostReverseProxy(url)
r.URL.Host = url.Host
r.URL.Scheme = url.Scheme
r.Header.Set("X-Forwarded-Host", r.Header.Get("Host"))
r.Host = url.Host
proxy.ServeHTTP(w, r)
and the error log show again
2020/05/26 15:49:29 http: proxy error: net/http: HTTP/1.x transport connection broken: http: ContentLength=32 with Body length 0
is the Clone() are not deep clone or i do wrong?
------------ this work------------
body, err := ioutil.ReadAll(r.Body)
if err != nil {
// ...
}
url, _ := url.Parse(config.GetGameHost())
r2 := r.Clone(r.Context())
r.Body = ioutil.NopCloser(bytes.NewReader(body))
r2.Body = ioutil.NopCloser(bytes.NewReader(body))
r.ParseForm()
proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(w, r2)
http.Request.Body can only be read once, a new body needs to be copied.
body,err := ioutil.ReadAll(r)
if err != nil {
// ...
}
r2 := r.Clone(r.Context())
// clone body
r.Body = ioutil.NopCloser(bytes.NewReader(body))
r2.Body = ioutil.NopCloser(bytes.NewReader(body))
// parse r1, proxy r2
r.ParseForm()
proxy.ServerHTTP(w, r2)
The body object defaults to net.Conn multi-layer encapsulation. Each time it uses the io.Reader interface to read 4kb, saving memory usage, it is generally used while reading from the network.
Therefore, the body can only be read from the network once. If you want the body object to be read repeatedly, you should read it all, save it, and use it.
I am making an http post request to a server using golang. Suppose the server is currently turned off (Means the machine on which the server runs is turned off) then the request is stuck at the IP layer. So my program execution is unable to proceed further. It is unable to proceed to the Application layer. So is there any way in golang to stop this.
I am using the following code.
req, err := http.NewRequest("POST", url, bytes.NewReader(b))
if err != nil {
return errors.Wrap(err, "new request error")
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return errors.Wrap(err, "http request error")
}
defer resp.Body.Close()
Is there anything that can be added to this, to terminate the request if it doesn't find anything from the IP layer.
The default http Client has no timeout. You can create an explicit http.Client yourself and set the timeout:
var cl = &http.Client{
Timeout: time.Second * 10,
}
resp, err := cl.Do(req)
if err != nil {
// err will be set on timeout
return errors.Wrap(err, "http request error")
}
defer resp.Body.Close()
If the server does not answer any more in the middle of a request, you can handle the timeout.
Use a non-default http.Transport with its DialContext field set to a function which uses a custom context with the properly configured timeout/deadline. Another option is to use a custom net.Dialer.
Something like this:
cli := http.Client{
Transport: &http.Transport{
DialContext: func (ctx context.Context, network, address string) (net.Conn, error) {
dialer := net.Dialer{
Timeout: 3 * time.Second,
}
return dialer.DialContext(ctx, network, address)
},
},
}
req, err := http.NewRequest(...)
resp, err := cli.Do(req)
Note that as per the net.Dialer's docs the context passed to its DialContext might trump the timeout set on the dialer itself—this is
exactly what we need: the dialer's Timeout field controls exactly the
"dialing" (TCP connection establishment) while you might also arm your
HTTP request with a context (using http.Request.WithContext) controlling
the timeout of the whole request, and also be able to cancel it at any time (including the dialing step).
Playground example.
The Transport #kostix refers to is definitely what you're looking for in this case. Transports as well as Clients are safe for concurrent use as well. But please read about the Transport (and I also advise reading about the Client as well) as there are a number of different ways to affect how you handle idle connections, not just the pre-mentioned DialContext.
As you may want to set your ResponseHeaderTimeout:
ResponseHeaderTimeout, if non-zero, specifies the amount of
time to wait for a server's response headers after fully
writing the request (including its body, if any). This
time does not include the time to read the response body.
Or, if you are using a secure connection, you may want to set your TLSHandshakeTimeout:
TLSHandshakeTimeout specifies the maximum amount of time waiting to
wait for a TLS handshake. Zero means no timeout.
For readability and maintainability, I suggest also maybe creating a function to build your Client, something along the lines of:
func buildClient(timeout time.Duration) *http.Client {
tr := &http.Transport{
IdleConnTimeout: timeout,
ResponseHeaderTimeout: timeout,
TLSHandshakeTimeout: timeout,
}
client := &http.Client{
Transport: tr,
Timeout: timeout,
}
return client
}