I'd like to parse the response of a web request, but I'm getting trouble accessing it as string.
func main() {
resp, err := http.Get("http://google.hu/")
if err != nil {
// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
ioutil.WriteFile("dump", body, 0600)
for i:= 0; i < len(body); i++ {
fmt.Println( body[i] ) // This logs uint8 and prints numbers
}
fmt.Println( reflect.TypeOf(body) )
fmt.Println("done")
}
How can I access the response as string? ioutil.WriteFile writes correctly the response to a file.
I've already checked the package reference but it's not really helpful.
bs := string(body) should be enough to give you a string.
From there, you can use it as a regular string.
A bit as in this thread
(updated after Go 1.16 -- Q1 2021 -- ioutil deprecation: ioutil.ReadAll() => io.ReadAll()):
var client http.Client
resp, err := client.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
bodyString := string(bodyBytes)
log.Info(bodyString)
}
See also GoByExample.
As commented below (and in zzn's answer), this is a conversion (see spec).
See "How expensive is []byte(string)?" (reverse problem, but the same conclusion apply) where zzzz mentioned:
Some conversions are the same as a cast, like uint(myIntvar), which just reinterprets the bits in place.
Sonia adds:
Making a string out of a byte slice, definitely involves allocating the string on the heap. The immutability property forces this.
Sometimes you can optimize by doing as much work as possible with []byte and then creating a string at the end. The bytes.Buffer type is often useful.
The method you're using to read the http body response returns a byte slice:
func ReadAll(r io.Reader) ([]byte, error)
official documentation
You can convert []byte to a string by using
body, err := ioutil.ReadAll(resp.Body)
bodyString := string(body)
Go 1.16+ update (February 2021)
Deprecation of io/ioutil
code should be
var client http.Client
resp, err := client.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
bodyBytes, err := io.ReadAll(resp.Body)
// if u want to read the body many time
// u need to restore
// reader := io.NopCloser(bytes.NewReader(bodyBytes))
if err != nil {
log.Fatal(err)
}
bodyString := string(bodyBytes)
log.Info(bodyString)
}
reference
https://golang.org/doc/go1.16#ioutil
https://stackoverflow.com/a/52076748/2876087
string(byteslice) will convert byte slice to string, just know that it's not only simply type conversion, but also memory copy.
Related
I'm trying to send form-data by making a post request. The api works fine (I've tested on postman), but I'm not sure why I'm having trouble to do it in golang. The form-data contains a task field and a file field. But if I do the following I get Bad Request. Any ideas why I might be getting this? Thanks in advance.
// Create new buffer and writer
buf := new(bytes.Buffer)
w := multipart.NewWriter(buf)
// read data from file
var fdata []byte
if fd, e := os.Open(pdf); e != nil {
log.Fatal(e)
} else {
fd.Read(fdata)
}
// create file field and write
part, err := w.CreateFormFile("file", pdf)
if err != nil {
log.Fatal(err)
}
part.Write(fdata)
// create the task field and write
part, err = w.CreateFormField("task")
if err != nil {
log.Fatal(err)
}
part.Write([]byte(os.Getenv("task")))
w.Close()
// Create a new request
req, err := http.NewRequest("POST", fmt.Sprintf("https://%v/v1/upload",os.Getenv("server")), buf)
// Set content type header
req.Header.Add("Content-Type", "multipart/form-data")
// Send the request
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
// other stuff
I receive the contents of a file from a data source in chunks. As and when I receive the chunk I want to send the chunk data to a service using http POST request. And by keeping alive the same http POST connection used for sending the first chunk I want to send the remaining chunks of data.
I came up with the following code snippet to implement something similar.
Server-Side
func handle(w http.ResponseWriter, req *http.Request) {
buf := make([]byte, 256)
var n int
for {
n, err := req.Body.Read(buf)
if n == 0 && err == io.EOF {
break
}
fmt.Printf(string(buf[:n]))
}
fmt.Printf(string(buf[:n]))
fmt.Printf("Transfer Complete")
}
Client-Side
type alphaReader struct {
reader io.Reader
}
func newAlphaReader(reader io.Reader) *alphaReader {
return &alphaReader{reader: reader}
}
func (a *alphaReader) Read(p []byte) (int, error) {
n, err := a.reader.Read(p)
return n, err
}
func (a *alphaReader) Reset(str string) {
a.reader = strings.NewReader(str)
}
func (a *alphaReader) Close() error {
return nil
}
func main() {
tr := http.DefaultTransport
alphareader := newAlphaReader(strings.NewReader("First Chunk"))
client := &http.Client{
Transport: tr,
Timeout: 0,
}
req := &http.Request{
Method: "POST",
URL: &url.URL{
Scheme: "http",
Host: "localhost:8080",
Path: "/upload",
},
ProtoMajor: 1,
ProtoMinor: 1,
ContentLength: -1,
Body: alphareader,
}
fmt.Printf("Doing request\n")
_, err := client.Do(req)
alphareader.Reset("Second Chunk")
fmt.Printf("Done request. Err: %v\n", err)
}
Here I want that when I do alphareader.Reset("Second Chunk"), the string "Second Chunk" should be sent using the POST connection made earlier. But that is not happening. The connection gets closed after sending the First Chunk of data. Also I have not written the Close() method properly which I'm not sure how to implement.
I'm newbie to golang and any suggestions would be greatly helpful regarding the same.
A *strings.Reader returns io.EOF after the initial string has been read and your wrapper does nothing to change that, so it cannot be reused. You're looking for io.Pipe to turn the request body into an io.Writer.
package main
import (
"io"
"net/http"
)
func main() {
pr, pw := io.Pipe()
req, err := http.NewRequest("POST", "http://localhost:8080/upload", pr)
if err != nil {
// TODO: handle error
}
go func() {
defer pw.Close()
if _, err := io.WriteString(pw, "first chunk"); err != nil {
_ = err // TODO: handle error
}
if _, err := io.WriteString(pw, "second chunk"); err != nil {
_ = err // TODO: handle error
}
}()
res, err := http.DefaultClient.Do(req)
if err != nil {
// TODO: handle error
}
res.Body.Close()
}
Also, don't initialize the request using a struct literal. Use one of the constructors instead. In your code you're not setting the Host and Header fields, for instance.
I want to construct a HTTP POST request with a body using Go net/http library.
The function I am using to construct the http request is the following: docs
http.NewRequest(method string, url string, body io.Reader)
I came up with 2 solutions, but I am trying to see which one is more idiomatic and extensible to support different body configurations.
Solution #1
bytesObj := []byte(`{"key":"value"}`)
body := bytes.NewBuffer(bytesObj)
Solution #2
bodyMap := map[string]string{"key":"value"}
bodyBytes, _ := json.Marshal(bodyMap)
body := bytes.NewBuffer(bodyBytes)
Ideally, I will move the code to a helper function that way I can customize the construction of the body. The helper function will be something like
func constructBody(someArgument) io.Reader {
return bodyHere
}
If the body is already string, options #1 is more compelling to me.
If you are only working with a key -> value with only string, option #2 is better.
But this will become cumbersome when you have nested struct
But most of the time in my experience we are dealing with struct. I like to make the struct closer to where the http call happened.
func main() {
ctx := context.Background()
body := struct {
Key string `json:"key"`
}{
Key: "value",
}
out, err := json.Marshal(body)
if err != nil {
log.Fatal(err)
}
req, err := http.NewRequest("POST", "http://example.com", bytes.NewBuffer(out))
if err != nil {
log.Fatal(err)
}
req = req.WithContext(ctx)
http.DefaultClient.Do(req)
}
And if the struct is used in multiple places, you can make a package level struct.
To have it more reusable, you could create a jsonReaderFactory, which just takes a few lines of code:
func jsonReaderFactory(in interface{}) (io.Reader, error) {
buf := bytes.NewBuffer(nil)
enc := json.NewEncoder(buf)
err := enc.Encode(in)
if err != nil {
return nil, fmt.Errorf("creating reader: error encoding data: %s", err)
}
return buf, nil
}
Example on playground
Hey there I would like to parse a http.resquest two times like below. When I parsed the Body the first time, the body will be closed. I need some help/hint what the best way is to handle this, do I have to create a copy of the request or is there a better way?
func myfunc(w http.ResponseWriter, req *http.Request) {
err := parseBody(req, &type1){
.....
}
err := parseBody(req, &type2){
.....
}
}
Thanks for help
It's true that you can read body only once and it's ok because to parse body more than once you don't have to read it more that one time. Let's consider simple example:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
)
type RequestData1 struct {
Code string `json:"code"`
Status string `json:"status"`
}
type RequestData2 struct {
Status string `json:"status"`
Message string `json:"message"`
}
func main() {
http.HandleFunc("/post", post)
http.ListenAndServe(":8080", nil)
}
If we use this code:
func post(w http.ResponseWriter, r *http.Request) {
body1, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
rd1 := RequestData1{}
err = json.Unmarshal(body1, &rd1)
if err != nil {
panic(err)
}
body2, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
rd2 := RequestData2{}
err = json.Unmarshal(body2, &rd2)
if err != nil {
panic(err) // panic!!!
}
fmt.Printf("rd1: %+v \nrd2: %+v", rd1, rd2)
w.WriteHeader(http.StatusOK)
w.Write([]byte(`Look into console.`))
}
we will have panic: http: panic serving [::1]:54581: unexpected end of JSON input
but with next code:
func post(w http.ResponseWriter, r *http.Request) {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
rd1 := RequestData1{}
err = json.Unmarshal(body, &rd1)
if err != nil {
panic(err)
}
rd2 := RequestData2{}
err = json.Unmarshal(body, &rd2)
if err != nil {
panic(err)
}
fmt.Printf("rd1: %+v \nrd2: %+v", rd1, rd2)
w.WriteHeader(http.StatusOK)
w.Write([]byte(`Look into console.`))
}
all works! You can test it by issuing request:
curl -X POST 'http://localhost:8080/post' \
-H 'Content-Type: application/json' -d '{"code":"200", "status": "OK", "message": "200 OK"}'
Result will be:
rd1: {Code:200 Status:OK}
rd2: {Status:OK Message:200 OK}
When you read request.Body, you're reading the stream from the client (e.g. web browser). The client only sends the request once. If you want to parse it multiple times, read the whole thing out into a buffer (e.g. a []byte) and then parse that as many times as you want. Just be mindful of the potential memory use of many concurrent requests with large payloads, as you'll be holding the full payload in memory at least until you're fully done parsing it.
I've been reading up on the various timeouts that are available on an http request and they all seem to act as hard deadlines on the total time of a request.
I am running an http download, I don't want to implement a hard timeout past the initial handshake as I don't know anything about my users connection and don't want to timeout on slow connections. What I would ideally like is to timeout after a period of inactivity (when nothing has been downloaded for x seconds). Is there any way to do this as a built in or do I have to interrupt based on stating the file?
The working code is a little hard to isolate but I think these are the relevant parts, there is another loop that stats the file to provide progress but I will need to refactor a bit to use this to interrupt the download:
// httspClientOnNetInterface returns an http client using the named network interface, (via proxy if passed)
func HttpsClientOnNetInterface(interfaceIP []byte, httpsProxy *Proxy) (*http.Client, error) {
log.Printf("Got IP addr : %s\n", string(interfaceIP))
// create address for the dialer
tcpAddr := &net.TCPAddr{
IP: interfaceIP,
}
// create the dialer & transport
netDialer := net.Dialer{
LocalAddr: tcpAddr,
}
var proxyURL *url.URL
var err error
if httpsProxy != nil {
proxyURL, err = url.Parse(httpsProxy.String())
if err != nil {
return nil, fmt.Errorf("Error parsing proxy connection string: %s", err)
}
}
httpTransport := &http.Transport{
Dial: netDialer.Dial,
Proxy: http.ProxyURL(proxyURL),
}
httpClient := &http.Client{
Transport: httpTransport,
}
return httpClient, nil
}
/*
StartDownloadWithProgress will initiate a download from a remote url to a local file,
providing download progress information
*/
func StartDownloadWithProgress(interfaceIP []byte, httpsProxy *Proxy, srcURL, dstFilepath string) (*Download, error) {
// start an http client on the selected net interface
httpClient, err := HttpsClientOnNetInterface(interfaceIP, httpsProxy)
if err != nil {
return nil, err
}
// grab the header
headResp, err := httpClient.Head(srcURL)
if err != nil {
log.Printf("error on head request (download size): %s", err)
return nil, err
}
// pull out total size
size, err := strconv.Atoi(headResp.Header.Get("Content-Length"))
if err != nil {
headResp.Body.Close()
return nil, err
}
headResp.Body.Close()
errChan := make(chan error)
doneChan := make(chan struct{})
// spawn the download process
go func(httpClient *http.Client, srcURL, dstFilepath string, errChan chan error, doneChan chan struct{}) {
resp, err := httpClient.Get(srcURL)
if err != nil {
errChan <- err
return
}
defer resp.Body.Close()
// create the file
outFile, err := os.Create(dstFilepath)
if err != nil {
errChan <- err
return
}
defer outFile.Close()
log.Println("starting copy")
// copy to file as the response arrives
_, err = io.Copy(outFile, resp.Body)
// return err
if err != nil {
log.Printf("\n Download Copy Error: %s \n", err.Error())
errChan <- err
return
}
doneChan <- struct{}{}
return
}(httpClient, srcURL, dstFilepath, errChan, doneChan)
// return Download
return (&Download{
updateFrequency: time.Microsecond * 500,
total: size,
errRecieve: errChan,
doneRecieve: doneChan,
filepath: dstFilepath,
}).Start(), nil
}
Update
Thanks to everyone who had input into this.
I've accepted JimB's answer as it seems like a perfectly viable approach that is more generalised than the solution I chose (and probably more useful to anyone who finds their way here).
In my case I already had a loop monitoring the file size so I threw a named error when this did not change for x seconds. It was much easier for me to pick up on the named error through my existing error handling and retry the download from there.
I probably crash at least one goroutine in the background with my approach (I may fix this later with some signalling) but as this is a short running application (its an installer) so this is acceptable (at least tolerable)
Doing the copy manually is not particularly difficult. If you're unsure how to properly implement it, it's only a couple dozen lines from the io package to copy and modify to suit your needs (I only removed the ErrShortWrite clause, because we can assume that the std library io.Writer implementations are correct)
Here is a copy work-alike function, that also takes a cancelation context and an idle timeout parameter. Every time there is a successful read, it signals to the cancelation goroutine to continue and start a new timer.
func idleTimeoutCopy(dst io.Writer, src io.Reader, timeout time.Duration,
ctx context.Context, cancel context.CancelFunc) (written int64, err error) {
read := make(chan int)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(timeout):
cancel()
case <-read:
}
}
}()
buf := make([]byte, 32*1024)
for {
nr, er := src.Read(buf)
if nr > 0 {
read <- nr
nw, ew := dst.Write(buf[0:nr])
written += int64(nw)
if ew != nil {
err = ew
break
}
}
if er != nil {
if er != io.EOF {
err = er
}
break
}
}
return written, err
}
While I used time.After for brevity, it's more efficient to reuse the Timer. This means taking care to use the correct reset pattern, as the return value of the Reset function is broken:
t := time.NewTimer(timeout)
for {
select {
case <-ctx.Done():
return
case <-t.C:
cancel()
case <-read:
if !t.Stop() {
<-t.C
}
t.Reset(timeout)
}
}
You could skip calling Stop altogether here, since in my opinion if the timer fires while calling Reset, it was close enough to cancel anyway, but it's often good to have the code be idiomatic in case this code is extended in the future.