I am using go http client to make get requests and the client has been initialised with a cookiejar however the response cookie array is empty. Has anyone got any idea what I am doing wrong?
jar, err := cookiejar.New(nil)
if err != nil {
log.Fatal(err)
}
s.http_client = &http.Client{Jar: jar}
resp, _ := s.http_client.Get(s.url)
fmt.Println(resp.Cookies()) returns an empty array although I can see cookies returned in firefox.
You create a cookiejar, and you can use it as seen in "how to follow location with cookie":
jar, err := cookiejar.New(&options)
if err != nil {
log.Fatal(err)
}
client := http.Client{Jar: jar} // <============
resp, err := client.Get("http://dubbelboer.com/302cookie.php")
if err != nil {
log.Fatal(err)
}
data, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
(introduced with Go1.1 as in this answer)
An http.Client struct has:
// Jar specifies the cookie jar.
// If Jar is nil, cookies are not sent in requests and ignored
// in responses.
Jar CookieJar
As 3of3 mentions, you don't need a cookiejar to fetch a cookie:
for _, cookie := range r.Cookies() {
fmt.Fprint(w, cookie.Name)
}
Check if the cookiejar is still empty after having read the full response body.
Related
I'm trying to send form-data by making a post request. The api works fine (I've tested on postman), but I'm not sure why I'm having trouble to do it in golang. The form-data contains a task field and a file field. But if I do the following I get Bad Request. Any ideas why I might be getting this? Thanks in advance.
// Create new buffer and writer
buf := new(bytes.Buffer)
w := multipart.NewWriter(buf)
// read data from file
var fdata []byte
if fd, e := os.Open(pdf); e != nil {
log.Fatal(e)
} else {
fd.Read(fdata)
}
// create file field and write
part, err := w.CreateFormFile("file", pdf)
if err != nil {
log.Fatal(err)
}
part.Write(fdata)
// create the task field and write
part, err = w.CreateFormField("task")
if err != nil {
log.Fatal(err)
}
part.Write([]byte(os.Getenv("task")))
w.Close()
// Create a new request
req, err := http.NewRequest("POST", fmt.Sprintf("https://%v/v1/upload",os.Getenv("server")), buf)
// Set content type header
req.Header.Add("Content-Type", "multipart/form-data")
// Send the request
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
// other stuff
When trying to perform a PUT request to a pre-signed Minio URL using the golang httpClient library the following error is returned:
<Error><Code>MissingContentLength</Code><Message>You must provide the Content-Length HTTP header.</Message><Key>obj</Key><BucketName>bucket</BucketName><Resource>/bucket/obj</Resource><RequestId>REMOVED</RequestId><HostId>REMOVED</HostId></Error>
I'm trying to upload a file to the URL created by running the following on a connected minioClient:
minioClient.PresignedPutObject(context.Background(), "bucket", "obj", time.Second*60)
The code which is erroring is:
url := "http://pre-signed-url-to-bucket-obj"
fileName := "test.txt"
file, err := os.Open(fileName)
if err != nil {
log.Fatal(err)
}
defer file.Close()
request, err := http.NewRequest(http.MethodPut, url, file)
if err != nil {
log.Fatal("Error creating request:", err)
}
// Tried including and excluding explicit Content-Length add, doesn't change response
// fStat, err := file.Stat()
// if err != nil {
// log.Fatal("Error getting file info:", err)
// }
// request.Header.Set("Content-Length", strconv.FormatInt(fStat.Size(), 10))
client := &http.Client{}
resp, err := client.Do(request)
if err != nil {
log.Fatal("Error performing request:", err)
}
defer resp.Body.Close()
content, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal("Error reading response:", err)
}
log.Println(string(content))
I've checked the Request and from what I'm able to tell Content-Length is being added.
A curl call with the --upload-file option specified will work:
curl -X PUT 'http://pre-signed-url-to-bucket-obj' --upload-file test.txt
I'm able to verify Content-Length is correctly added.
I would like to avoid a form as it does weird stuff to the obj on Minio's end.
Any help is much appreciated!
Do this to explicit content-length explicitly:
request.ContentLength = fStat.Size()
I verified that the above code works with this fix
I observed that Content-Length header is not getting set for PATCH requests with empty/nil payload. Even if we manually set it by req.Header.Set("content-length", "0") it is not actually getting set in the out going request.
This strange behaviour (Go bug?) happens only for PATCH requests and only when the payload is empty or nil (or set to http.NoBody)
package main
import (
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
url := "http://localhost:9999"
method := "PATCH"
payload := strings.NewReader("")
client := &http.Client {
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Authorization", "Bearer my-token")
req.Header.Set("Content-Length", "0") //this is not honoured
res, err := client.Do(req)
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
fmt.Println(string(body))
}
This is reproducible even in the latest go version 1.15.
Just run the above code against a simple http server and see for yourself.
Is there any solution/workaround to send a PATCH request with Content-Length set to 0 ?
You can tell the HTTP client to include a Content-Length header with value 0 by setting TransferEncoding to identity as follows:
url := "http://localhost:9999"
method := "PATCH"
client := &http.Client{}
req, err := http.NewRequest(method, url, http.NoBody)
if err != nil {
panic(err)
}
req.TransferEncoding = []string{"identity"}
req.Header.Set("Authorization", "Bearer my-token")
// req.Header.Set("Content-Length", "0")
Note the following changes to your original code:
the important one: req.TransferEncoding = []string{"identity"}
the idiomatic way of specifying an empty body: http.NoBody (no impact on sending the length)
commented out req.Header.Set("Content-Length", "0"), the client fills it in by itself
also changed to panic on an error, you probably don't want to continue
The transfer encoding of identity is not written to the request, so except for the header Content-Length = 0, the request looks the same as before.
This is unfortunately not documented (feel free to file an issue with the Go team), but can be seen in the following code:
The tedious details:
transferWriter.writeHeader checks the following to write the Content-Length header:
// Write Content-Length and/or Transfer-Encoding whose values are a
// function of the sanitized field triple (Body, ContentLength,
// TransferEncoding)
if t.shouldSendContentLength() {
if _, err := io.WriteString(w, "Content-Length: "); err != nil {
return err
}
if _, err := io.WriteString(w, strconv.FormatInt(t.ContentLength, 10)+"\r\n"); err != nil {
return err
}
In turn, shouldCheckContentLength looks at the transfer encoding in case of zero length:
if t.ContentLength == 0 && isIdentity(t.TransferEncoding) {
if t.Method == "GET" || t.Method == "HEAD" {
return false
}
return true
}
The isIdentity verifies that TransferEncoding is exactly []string{"identity"}:
func isIdentity(te []string) bool { return len(te) == 1 && te[0] == "identity" })
Go version: go1.8.1 windows/amd64
Sample code for HTTP request is:
func (c *Client) RoundTripSoap12(action string, in, out Message) error {
fmt.Println("****************************************************************")
headerFunc := func(r *http.Request) {
r.Header.Add("Content-Type", fmt.Sprintf("text/xml; charset=utf-8"))
r.Header.Add("SOAPAction", fmt.Sprintf(action))
r.Cookies()
}
return doRoundTrip(c, headerFunc, in, out)
}
func doRoundTrip(c *Client, setHeaders func(*http.Request), in, out Message) error {
req := &Envelope{
EnvelopeAttr: c.Envelope,
NSAttr: c.Namespace,
Header: c.Header,
Body: Body{Message: in},
}
if req.EnvelopeAttr == "" {
req.EnvelopeAttr = "http://schemas.xmlsoap.org/soap/envelope/"
}
if req.NSAttr == "" {
req.NSAttr = c.URL
}
var b bytes.Buffer
err := xml.NewEncoder(&b).Encode(req)
if err != nil {
return err
}
cli := c.Config
if cli == nil {
cli = http.DefaultClient
}
r, err := http.NewRequest("POST", c.URL, &b)
if err != nil {
return err
}
setHeaders(r)
if c.Pre != nil {
c.Pre(r)
}
fmt.Println("*************", r)
resp, err := cli.Do(r)
if err != nil {
fmt.Println("error occured is as follows ", err)
return err
}
fmt.Println("response headers are: ", resp.Header.Get("sprequestguid"))
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
// read only the first Mb of the body in error case
limReader := io.LimitReader(resp.Body, 1024*1024)
body, _ := ioutil.ReadAll(limReader)
return fmt.Errorf("%q: %q", resp.Status, body)
}
return xml.NewDecoder(resp.Body).Decode(out)
I will call the RoundTripSoap12 function on the corresponding HTTP client.
When I send a request for the first time I will be getting some headers in the HTTP response, so these HTTP response headers should be sent as-is in my next HTTP request.
You may be interested in the httputil package and the reverse proxy example provided if you wish to proxy requests transparently:
https://golang.org/src/net/http/httputil/reverseproxy.go
You can copy the headers from one request to another one fairly easily - the Header is a separate object, if r and rc are http.Requests and you don't mind them sharing a header (you may need to clone instead if you want independent requests):
rc.Header = r.Header // note shallow copy
fmt.Println("Headers", r.Header, rc.Header)
https://play.golang.org/p/q2KUHa_qiP
Or you can look through keys and values and only copy certain headers, and/or do a clone instead to ensure you share no memory. See the http util package here for examples of this - see the functions cloneHeader and copyHeader inside reverseproxy.go linked above.
I'm working on an api endpoint in go that will accept an upload and then immediately forward to another API. I don't want to write the file to disk anywhere, but I'm not sure storing the file temporarily in memory the way I have is correct either. All the examples that I can find deal with saving the file to disk. I've posted what I'm doing below. The response I get back from the second API is that I failed to post a file, but I can see that it is receiving the "userID" field. Can someone please point out what I'm doing wrong as well as possibly advise if this is the best way to go about this?
Route Handler
func (r *Routes) forwardFile(w http.ResponseWriter, req *http.Request){
parameters := mux.Vars(req)
userID := parameters["userID"]
const maxFileSize = 1 * 1024 * 1024 // 1MB
// pull in the uploaded file into memory
req.ParseMultipartForm(maxFileSize)
file, fileHeader, err := req.FormFile("fileUpload")
if err != nil {
encodeResponse(w, req, response{obj: nil, err: err})
return
}
defer file.Close()
success, err := service.DoForwardFile(userID, file, fileHeader)
encodeResponse(w, req, response{obj: success, err: err})
}
Service Handler
func (b *base) DoForwardFile(userID int, file multipart.File, fileHeader *multipart.FileHeader) (FileForwardedResponse, error) {
// start building our request to forward the file
var resp *http.Response
defer func() {
if resp != nil {
resp.Body.Close()
}
reportStat.Complete(0)
}()
// build a form body
body := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(body)
// add form fields
bodyWriter.WriteField("userID", userID)
// add a form file to the body
fileWriter, err := bodyWriter.CreateFormFile("fileUpload", fileHeader.Filename)
if err != nil {
return FileForwardedResponse{}, err
}
// copy the file into the fileWriter
_, err = io.Copy(fileWriter, file)
if err != nil {
return FileForwardedResponse{}, err
}
// Close the body writer
bodyWriter.Close()
// build request url
apiURL := fmt.Sprintf("%s/v2/users/%d/files", config.APIURL, userID)
// send request
client := &http.Client{Timeout: time.Second * 10}
req, err := http.NewRequest("POST", apiURL, body)
resp, err = client.Do(req)
...
}
You're not setting the Content-Type for the request. Even if the header gets set automatically to multipart/form-data, it's missing the data boundary.
req, err := http.NewRequest("POST", uri, body)
if err != nil {
return FileForwardedResponse{}, err
}
req.Header.Set("Content-Type", bodyWriter.FormDataContentType())
...