Hey there I would like to parse a http.resquest two times like below. When I parsed the Body the first time, the body will be closed. I need some help/hint what the best way is to handle this, do I have to create a copy of the request or is there a better way?
func myfunc(w http.ResponseWriter, req *http.Request) {
err := parseBody(req, &type1){
.....
}
err := parseBody(req, &type2){
.....
}
}
Thanks for help
It's true that you can read body only once and it's ok because to parse body more than once you don't have to read it more that one time. Let's consider simple example:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
)
type RequestData1 struct {
Code string `json:"code"`
Status string `json:"status"`
}
type RequestData2 struct {
Status string `json:"status"`
Message string `json:"message"`
}
func main() {
http.HandleFunc("/post", post)
http.ListenAndServe(":8080", nil)
}
If we use this code:
func post(w http.ResponseWriter, r *http.Request) {
body1, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
rd1 := RequestData1{}
err = json.Unmarshal(body1, &rd1)
if err != nil {
panic(err)
}
body2, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
rd2 := RequestData2{}
err = json.Unmarshal(body2, &rd2)
if err != nil {
panic(err) // panic!!!
}
fmt.Printf("rd1: %+v \nrd2: %+v", rd1, rd2)
w.WriteHeader(http.StatusOK)
w.Write([]byte(`Look into console.`))
}
we will have panic: http: panic serving [::1]:54581: unexpected end of JSON input
but with next code:
func post(w http.ResponseWriter, r *http.Request) {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
rd1 := RequestData1{}
err = json.Unmarshal(body, &rd1)
if err != nil {
panic(err)
}
rd2 := RequestData2{}
err = json.Unmarshal(body, &rd2)
if err != nil {
panic(err)
}
fmt.Printf("rd1: %+v \nrd2: %+v", rd1, rd2)
w.WriteHeader(http.StatusOK)
w.Write([]byte(`Look into console.`))
}
all works! You can test it by issuing request:
curl -X POST 'http://localhost:8080/post' \
-H 'Content-Type: application/json' -d '{"code":"200", "status": "OK", "message": "200 OK"}'
Result will be:
rd1: {Code:200 Status:OK}
rd2: {Status:OK Message:200 OK}
When you read request.Body, you're reading the stream from the client (e.g. web browser). The client only sends the request once. If you want to parse it multiple times, read the whole thing out into a buffer (e.g. a []byte) and then parse that as many times as you want. Just be mindful of the potential memory use of many concurrent requests with large payloads, as you'll be holding the full payload in memory at least until you're fully done parsing it.
Related
I am writing an API whichs has to redirect incoming requests to another service, the response must be forwarded to the original requester.
I figured a simple function like below should do the trick, but I was wrong.
I receive the data from my redirected response, however when I send it back to the initial request I receive this response without any data Could not get response. Error: socket hang up
If I try to execute the very same request using postman straight to the redirect URL it works perfectly fine.
func initialAssetsHandler(w http.ResponseWriter, r *http.Request) {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
fmt.Println(err)
return
}
resp, err := http.Post(conf.redirectURL, "application/json", bytes.NewReader(body))
if err != nil {
log.Error(err)
}
defer resp.Body.Close()
buf := new(bytes.Buffer)
buf.ReadFrom(resp.Body)
log.Info(string(buf.Bytes()))
var data json.RawMessage
if err = json.NewDecoder(resp.Body).Decode(&data); err != nil {
fmt.Println(err)
return
}
helper.SendJsonRaw(w, 200, data)
}
Here is the SendJsonRaw function:
func SendJsonRaw(w http.ResponseWriter, status int, r json.RawMessage) error {
w.Header().Set(HeaderContentType, MimeApplicationJSON)
w.WriteHeader(status)
_, err := w.Write(r)
return err
}
The r.Body is read by the json decoder up to EOF, then when you pass it to the redirect request it looks empty to the http.Client and therefore it sends no body. You need to retain the content of the body.
For example you can do the following:
func initialAssetsHandler(w http.ResponseWriter, r *http.Request) {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
fmt.Println(err)
return
}
var initialAssets TagAssets
if err := json.Unmarshal(&initialAssets, body); err != nil {
if !strings.Contains(err.Error(), "json: invalid use of ,string struct tag, trying to unmarshal") {
helper.SendJsonError(w, http.StatusBadRequest, err)
return
}
}
resp, err := http.Post(conf.redirectURL, "application/json", bytes.NewReader(body))
if err != nil {
log.Error(err)
}
defer resp.Body.Close()
log.Info(resp)
var data json.RawMessage
if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
fmt.Println(err)
return
}
helper.SendJsonOk(w, data)
}
I receive the contents of a file from a data source in chunks. As and when I receive the chunk I want to send the chunk data to a service using http POST request. And by keeping alive the same http POST connection used for sending the first chunk I want to send the remaining chunks of data.
I came up with the following code snippet to implement something similar.
Server-Side
func handle(w http.ResponseWriter, req *http.Request) {
buf := make([]byte, 256)
var n int
for {
n, err := req.Body.Read(buf)
if n == 0 && err == io.EOF {
break
}
fmt.Printf(string(buf[:n]))
}
fmt.Printf(string(buf[:n]))
fmt.Printf("Transfer Complete")
}
Client-Side
type alphaReader struct {
reader io.Reader
}
func newAlphaReader(reader io.Reader) *alphaReader {
return &alphaReader{reader: reader}
}
func (a *alphaReader) Read(p []byte) (int, error) {
n, err := a.reader.Read(p)
return n, err
}
func (a *alphaReader) Reset(str string) {
a.reader = strings.NewReader(str)
}
func (a *alphaReader) Close() error {
return nil
}
func main() {
tr := http.DefaultTransport
alphareader := newAlphaReader(strings.NewReader("First Chunk"))
client := &http.Client{
Transport: tr,
Timeout: 0,
}
req := &http.Request{
Method: "POST",
URL: &url.URL{
Scheme: "http",
Host: "localhost:8080",
Path: "/upload",
},
ProtoMajor: 1,
ProtoMinor: 1,
ContentLength: -1,
Body: alphareader,
}
fmt.Printf("Doing request\n")
_, err := client.Do(req)
alphareader.Reset("Second Chunk")
fmt.Printf("Done request. Err: %v\n", err)
}
Here I want that when I do alphareader.Reset("Second Chunk"), the string "Second Chunk" should be sent using the POST connection made earlier. But that is not happening. The connection gets closed after sending the First Chunk of data. Also I have not written the Close() method properly which I'm not sure how to implement.
I'm newbie to golang and any suggestions would be greatly helpful regarding the same.
A *strings.Reader returns io.EOF after the initial string has been read and your wrapper does nothing to change that, so it cannot be reused. You're looking for io.Pipe to turn the request body into an io.Writer.
package main
import (
"io"
"net/http"
)
func main() {
pr, pw := io.Pipe()
req, err := http.NewRequest("POST", "http://localhost:8080/upload", pr)
if err != nil {
// TODO: handle error
}
go func() {
defer pw.Close()
if _, err := io.WriteString(pw, "first chunk"); err != nil {
_ = err // TODO: handle error
}
if _, err := io.WriteString(pw, "second chunk"); err != nil {
_ = err // TODO: handle error
}
}()
res, err := http.DefaultClient.Do(req)
if err != nil {
// TODO: handle error
}
res.Body.Close()
}
Also, don't initialize the request using a struct literal. Use one of the constructors instead. In your code you're not setting the Host and Header fields, for instance.
I am trying to get the parameters made in a POST request, but I am not able to make it, my code is:
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
http.HandleFunc("/", hello)
fmt.Printf("Starting server for testing HTTP POST...\n")
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatal(err)
}
}
func hello(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
http.Error(w, "404 not found.", http.StatusNotFound)
return
}
switch r.Method {
case "POST":
// Call ParseForm() to parse the raw query and update r.PostForm and r.Form.
if err := r.ParseForm(); err != nil {
fmt.Fprintf(w, "ParseForm() err: %v", err)
return
}
name := r.Form.Get("name")
age := r.Form.Get("age")
fmt.Print("This have been received:")
fmt.Print("name: ", name)
fmt.Print("age: ", age)
default:
fmt.Fprintf(w, "Sorry, only POST methods are supported.")
}
}
I am making the POST request in the terminal as follows:
curl -X POST -d '{"name":"Alex","age":"50"}' localhost:8080
And then the output is:
This have been received:name: age:
Why it is not taking the parameters? What I am doing wrong?
As you pass your body as a json object, you better define a Go struct matching that object and decode the request body to the object.
type Info struct {
Name string
Age int
}
info := &Info{}
if err := json.NewDecoder(r.Body).Decode(info); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
_ = json.NewEncoder(w).Encode(info)
You can find the whole working code here.
$ curl -X POST -d '{"name":"Alex","age":50}' localhost:8080
This POST request is working fine now.
You could modify the Go struct and also the response object as you like .
I'm trying to create a simple http service with the endpoint to download file to the local system in Go. The link comes in ?uri tag, but when I want to get it I receive an empty string. I tried to parse the form of my request but it didn't help. Here is my code:
func main() {
http.HandleFunc("/download", DownloadHandler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func DownloadHandler(writer http.ResponseWriter, request *http.Request) {
prsErr := request.ParseForm()
if prsErr != nil{
panic(prsErr)
}
uri := request.FormValue("?uri")
_, _ = writer.Write([]byte(uri))
err := DownloadFile("img.png", uri)
if err != nil {
panic(err)
}
}
func DownloadFile(filepath string, url string) error {
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Write the body to file
_, err = io.Copy(out, resp.Body)
if err != nil {
return err
}
return nil
}
I will appreciate any help! Thank you!
invalid at request.FormValue("?uri")
uri := request.FormValue("uri")
I'm writing a simple web app in Go and I want my responses to be streamed to the client (i.e. not buffered and sent in blocks once the request is fully processed) :
func handle(res http.ResponseWriter, req *http.Request) {
fmt.Fprintf(res, "sending first line of data")
sleep(10) //not real code
fmt.Fprintf(res, "sending second line of data")
}
From the client point of view, the two lines will be sent at the same time. Any suggestions are appreciated :)
Edit after #dystroy answer
It's possible to flush after each write I personally make, but in my use case it's not enough:
cmd := exec.Command("a long command that outputs lots of lines")
cmd.Stdout = res //where res is a http.ResponseWritter
cmd.Stderr = res
err := cmd.Run()
I want the output of my cmd to be flushed as well. Anyway to "autoflush" the ResponseWritter ?
Solution
I found help on golang's mailing list. There is 2 way to achieve this: using hijacker that allow to take over the underlying TCP connection of HTTP, or piping the stdout and stderr of the command in a go routine that will write and flush :
pipeReader, pipeWriter := io.Pipe()
cmd.Stdout = pipeWriter
cmd.Stderr = pipeWriter
go writeCmdOutput(res, pipeReader)
err := cmd.Run()
pipeWriter.Close()
//---------------------
func writeCmdOutput(res http.ResponseWriter, pipeReader *io.PipeReader) {
buffer := make([]byte, BUF_LEN)
for {
n, err := pipeReader.Read(buffer)
if err != nil {
pipeReader.Close()
break
}
data := buffer[0:n]
res.Write(data)
if f, ok := res.(http.Flusher); ok {
f.Flush()
}
//reset buffer
for i := 0; i < n; i++ {
buffer[i] = 0
}
}
}
Last update
Even nicer: http://play.golang.org/p/PpbPyXbtEs
As implied in the documentation, some ResponseWriter may implement the Flusher interface.
This means you can do something like this :
func handle(res http.ResponseWriter, req *http.Request) {
fmt.Fprintf(res, "sending first line of data")
if f, ok := res.(http.Flusher); ok {
f.Flush()
} else {
log.Println("Damn, no flush");
}
sleep(10) //not real code
fmt.Fprintf(res, "sending second line of data")
}
Be careful that buffering can occur in many other places in the network or client side.
Sorry if I've misunderstood your question, but would something like the below do the trick?
package main
import (
"bytes"
"fmt"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
body := make([]byte, int(r.ContentLength))
b := bytes.NewBuffer(body)
if _, err := b.ReadFrom(r.Body); err != nil {
fmt.Fprintf(w, "%s", err)
}
if _, err := b.WriteTo(w); err != nil {
fmt.Fprintf(w, "%s", err)
}
}
func main() {
http.HandleFunc("/", handler)
if err := http.ListenAndServe(":8080", nil); err != nil {
panic(err)
}
}
$ curl --data "param1=value1¶m2=value2" http://localhost:8080
returns:
param1=value1¶m2=value2
You could always append whatever data you wanted to body, or read more bytes into the buffer from elsewhere before writing it all out again.