Note: All error handling code is omitted for simplicity. I handled them locally, and no errors were produced.
In Golang, I tried to use the code below to read from a http.Request.Body from a POST request:
func readBody(req *http.Request) string {
bytes, _ := httputils.DumpRequestOut(req, true)
return string(bytes)
}
It shows a non-zero Content-Length, but no content returned:
ContentLength=413 with Body length 0
. I tried with the code below, also no luck:
func readBody(req *http.Request) string {
bytes, _ := ioutil.ReadAll(req.Body)
return string(bytes)
}
It returns an empty string. After googling, I found a blog about this problem: Golang: Read from an io.ReadWriter without losing its content. I tried to follow the pattern, still no luck:
func readBody(req *http.Request) string {
bodyBytes, _ := ioutil.ReadAll(req.Body)
// Restore the io.ReadCloser to its original state
req.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))
// Use the content
return string(bodyBytes)
}
Any suggestions? Thanks in advance :)
If the request does not include an appropriate "Content-Type" header you may see a 0 length body when you try to read it.
This is happening because you call httputils.DumpRequestOut(req, true) after the http client has performed the request and the body is drained.
Make sure you do the following step in order:
httputils.DumpRequestOut(req, true)
resp, err :=http.DefaultClient.Do(req)
httputil.DumpResponse(res, true)
Related
I am trying to retrieve the filename from this http writer for testing purposes.
On the server I have:
func servefile(w http.ResponseWriter, r *http.Request ) {
...
// file.Name() is randomized with os.CreateTemp(dir, temp+"*"+ext) above
w.Header().Set("Content-Disposition", "attachment; filename="+file.Name())
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeFile(w, r, file.Name()+".xyz") // serve file to user to download
...
}
*I put .xyz as a place holder for this demonstration
I am testing this function programatically with Go and I want to access the filename to able to save it in a variable in the client code.
I have looked at this post How can I parse the Content-Disposition header to retrieve the filename property? , but I have not gotten it to work. I have no clue what the filename is on the client side so I don't know how to reference it specifically. I know my code on the server side works, because when I send a request (through the browser) to this endpoint/function, the "Downloads" popup shows the file download progress with the name of the file.
EDIT** This is the client code I am calling it from:
func TestGetFile(t *testing.T) {
...
cid := "some string"
// requestfile() creates, executes, and returns an httptest.ResponseRecorder to the requestFile endpoint
reqfileRespRecorder := requestfile()
// createTmpFile creates a new file out of the contents recieved in requestfile()
filePath := "/tmp/temp.xyz"
file := createTmpFile(reqfileRespRecorder , filePath)
// CreateWriter() - writes file contents to body of multipart.Writer
w, body := createWriter(file)
// Create request to postRecord endpoint
req, err := http.NewRequest("POST", "/PostRecord?CID="+cid, body)
check(err)
req.Header.Add("Content-Type", w.FormDataContentType())
// execute request to PostRecord endpoint. returns an httptest.ResponseRecorder
respRecorder := executeRequest(PostRecord, req)
disposition, params, err := mime.ParseMediaType(`Content-Disposition`)
...
}
Based on #BaytaDarrell's comment. It dawned on me that I could print out the responses. That helped me realize that I was trying to find the content-disposition after the wrong request/response. The linked post still didn't help, but I got my code working like this:
func TestGetFile(t *testing.T) {
...
cid := "some string"
// requestfile() creates, executes, and returns an httptest.ResponseRecorder to the requestFile endpoint
reqfileRespRecorder := requestfile()
disp := reqfileRespRecorder.Header().Get("Content-Disposition")
line := strings.Split(disp, "=")
filename := line[1]
fmt.Println("filename: ", filename)
// createTmpFile creates a new file out of the contents recieved in requestfile()
filePath := "/tmp/temp.xyz"
file := createTmpFile(reqfileRespRecorder , filePath)
// CreateWriter() - writes file contents to body of multipart.Writer
w, body := createWriter(file)
...
}
Their comment realized I should re-look at the httptest package documentation. Here I found the Header() function and that I can use it to look at the header with Get().
This line reqfileRespRecorder.Header().Get("Content-Disposition") returns attachment; filename=temp37bf73gd.xyz and to store the filename in a variable I split on =.
I'm trying to extract a value from an HTTP POST request body (in my simple Go HTTP server) using the net/http PostFormValue and my output is an empty string when I'm looking for the any key in general, but in my case trying to fetch the hub.secret for use in a HMAC check. I use Postman to send the request to my localhost:8080 instance using the Gorilla/mux router, with header Content-Type: application/x-www-form-urlencoded set.
My handler looks like so:
func rootPostHandler(w http.ResponseWriter, r *http.Request) {
var expectedMac []byte
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("r.Body is:", string(body)) // debug: print the request POST body
message := body // debug: set message just for extra clarity
errParse := r.ParseForm()
if errParse != nil {
// handle err
}
secret := []byte(r.PostFormValue("hub.secret"))
log.Println("secret is: ", string(secret))
mac := hmac.New(sha256.New, secret)
mac.Write(message)
expectedMac = mac.Sum(nil)
fmt.Println("Is HMAC equal? ", hmac.Equal(message, expectedMac))
w.Header().Add("X-Hub-Signature", "sha256="+string(message))
}
The r.Body:
hub.callback=http%253A%252F%252Fweb-sub-client%253A8080%252FbRxvcmOcNk&hub.mode=subscribe&hub.secret=xTgSGLOtPNrBLLgYcKnL&hub.topic=%252Fa%252Ftopic
And the output for print secret etc is empty string, meaning it can't find hub.secret, right? What am I missing here?
The application reads the request body to EOF on this line:
body, err := ioutil.ReadAll(r.Body)
ParseForm returns an empty form because the body is at EOF at this line:
errParse := r.ParseForm()
The request body is read from the network connection. The request body cannot be read a second time.
Remove the call to ioutil.ReadAll or create a new body reader using the data returned from ioutil.ReadAll:
r.Body = io.NopCloser(bytes.NewReader(body))
Below handler handles GET request, without populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Below handler handles GET request, populating http Response header:
// ListAll handles GET requests and returns all current products
func (p *ProductHandler) ListAll(rw http.ResponseWriter, r *http.Request) {
p.l.Println("[DEBUG] get all records")
rw.Header().Add("Content-Type", "application/json")
prods := data.GetProducts()
err := data.ToJSON(prods, rw)
if err != nil {
// we should never be here but log the error just incase
p.l.Println("[ERROR] serializing product", err)
}
}
Both cases are working fine with simple curl request.
For any http client,
When do we need to populate content-type header, before sending the response, to client?
Always read the documentation first!
The answer to this is clearly covered here (emphasis obviously added):
// If WriteHeader has not yet been called, Write calls
// WriteHeader(http.StatusOK) before writing the data. If the Header
// does not contain a Content-Type line, Write adds a Content-Type set
// to the result of passing the initial 512 bytes of written data to
// DetectContentType. Additionally, if the total size of all written
// data is under a few KB and there are no Flush calls, the
// Content-Length header is added automatically.
To explicitly answer your secondary question:
When do we need to populate content-type header?
Any time you don't want it to be automatically detected. Automatic detection is imprecise, so you generally don't want to rely on it.
consider this scenario!
after successful execution of a http request, what if there is an error while performing json encoding, how to override the header code
func writeResp(w http.ResponseWriter, code int, data interface{}) {
w.Header().Set("Content-Type", "application/json")
//Here I set the status to 201 StatusCreated
w.WriteHeader(code)
s := success{Data: data}
//what if there is an error here and want to override the status to 5xx error
//how to handle error here, panic?, http.Error() is not an option because as we already wrote header to 201, it just prints `http: multiple response.WriteHeader calls`
if err := json.NewEncoder(w).Encode(s); err != nil {
w.Header().Set("Content-Type", "application/json")
//it throws http: multiple response.WriteHeader calls here as we already wrote header above to 201
w.WriteHeader(code)
e := errorResponse{
Code: code,
Error: error,
Description: msg,
}
if err := json.NewEncoder(w).Encode(e); err != nil {
//same how to handle here
}
}
}
I have multiple options here, if we do just fatal logging the user won't know exactly what happened, even if I write string using w.Write([]byte(msg)) still the status says 201 created, how to respond with error code 5xx
any help is greatly appreciated
First of all, it does not seem very likely that you get an error when encoding.
See this question for reasons for Marshal to fail:
What input will cause golang's json.Marshal to return an error?
The other potential cause of error would be some problem with actually writing the data to the response stream, but in that case you'd not be able to write your custom error either.
Going back to your question, if you are concerned that encoding your object might fail, you can first Marshal your data (checking for error), then only write the 201 status code (and the encoded data) if marshalling succeeded.
Modifying your example a bit:
s := success{Data: data}
jsonData, err := json.Marshal(s)
if err != nil {
// write your error to w, then return
}
w.WriteHeader(code)
w.Header().Set("Content-Type", "application/json")
w.Write(jsonData)
Now, that last write can also throw an error.
But if that happens, it will also fail when writing your custom error, so in that case you'd better log that in the server side (or send that error to a tracker such as New Relic, etc).
I am using Go to make an HTTP GET request to an external web service. For some reason, the body of the response is always empty; the content length is always zero bytes. The response status code is always 200, however, and the call to Client.Do returns no error. The request requires an Authorization header, so I am using the http.NewRequest / http.Client.Do pattern to submit the request, as you'll see below. I have done requests similar to these in the past, but never using a GET that required a header. It seems unlikely that this the cause, but I wonder if it may be related. If anyone can spot any potential issues with the pattern used or perhaps has had a similar experience, I'd really appreciate any help.
Thank you.
if req, err := http.NewRequest("GET", "https://api.molt.in/v1/orders/11111111/items", nil); err != nil {
return nil, err
} else {
client := &http.Client{}
req.Header.Add("Authorization", "secretToken")
if resp, err := client.Do(req); err != nil {
return nil, err
} else {
defer resp.Body.Close()
return readBody(resp.Body)
}
}
I finally discovered the source of the problem. It had nothing to do with the request being made, or the response being received. It had to do with the parsing of the response.
I was using bufio.NewScanner.Text to attempt to convert the response body into a string. Replacing this call with one to ioutil.ReadAll output the string that I originally expected.
Thanks for all of your help, and apologies for the misleading question.