Http POST request with body Go - http

I want to construct a HTTP POST request with a body using Go net/http library.
The function I am using to construct the http request is the following: docs
http.NewRequest(method string, url string, body io.Reader)
I came up with 2 solutions, but I am trying to see which one is more idiomatic and extensible to support different body configurations.
Solution #1
bytesObj := []byte(`{"key":"value"}`)
body := bytes.NewBuffer(bytesObj)
Solution #2
bodyMap := map[string]string{"key":"value"}
bodyBytes, _ := json.Marshal(bodyMap)
body := bytes.NewBuffer(bodyBytes)
Ideally, I will move the code to a helper function that way I can customize the construction of the body. The helper function will be something like
func constructBody(someArgument) io.Reader {
return bodyHere
}

If the body is already string, options #1 is more compelling to me.
If you are only working with a key -> value with only string, option #2 is better.
But this will become cumbersome when you have nested struct
But most of the time in my experience we are dealing with struct. I like to make the struct closer to where the http call happened.
func main() {
ctx := context.Background()
body := struct {
Key string `json:"key"`
}{
Key: "value",
}
out, err := json.Marshal(body)
if err != nil {
log.Fatal(err)
}
req, err := http.NewRequest("POST", "http://example.com", bytes.NewBuffer(out))
if err != nil {
log.Fatal(err)
}
req = req.WithContext(ctx)
http.DefaultClient.Do(req)
}
And if the struct is used in multiple places, you can make a package level struct.

To have it more reusable, you could create a jsonReaderFactory, which just takes a few lines of code:
func jsonReaderFactory(in interface{}) (io.Reader, error) {
buf := bytes.NewBuffer(nil)
enc := json.NewEncoder(buf)
err := enc.Encode(in)
if err != nil {
return nil, fmt.Errorf("creating reader: error encoding data: %s", err)
}
return buf, nil
}
Example on playground

Related

Sending data in Chunks using single HTTP Post connection

I receive the contents of a file from a data source in chunks. As and when I receive the chunk I want to send the chunk data to a service using http POST request. And by keeping alive the same http POST connection used for sending the first chunk I want to send the remaining chunks of data.
I came up with the following code snippet to implement something similar.
Server-Side
func handle(w http.ResponseWriter, req *http.Request) {
buf := make([]byte, 256)
var n int
for {
n, err := req.Body.Read(buf)
if n == 0 && err == io.EOF {
break
}
fmt.Printf(string(buf[:n]))
}
fmt.Printf(string(buf[:n]))
fmt.Printf("Transfer Complete")
}
Client-Side
type alphaReader struct {
reader io.Reader
}
func newAlphaReader(reader io.Reader) *alphaReader {
return &alphaReader{reader: reader}
}
func (a *alphaReader) Read(p []byte) (int, error) {
n, err := a.reader.Read(p)
return n, err
}
func (a *alphaReader) Reset(str string) {
a.reader = strings.NewReader(str)
}
func (a *alphaReader) Close() error {
return nil
}
func main() {
tr := http.DefaultTransport
alphareader := newAlphaReader(strings.NewReader("First Chunk"))
client := &http.Client{
Transport: tr,
Timeout: 0,
}
req := &http.Request{
Method: "POST",
URL: &url.URL{
Scheme: "http",
Host: "localhost:8080",
Path: "/upload",
},
ProtoMajor: 1,
ProtoMinor: 1,
ContentLength: -1,
Body: alphareader,
}
fmt.Printf("Doing request\n")
_, err := client.Do(req)
alphareader.Reset("Second Chunk")
fmt.Printf("Done request. Err: %v\n", err)
}
Here I want that when I do alphareader.Reset("Second Chunk"), the string "Second Chunk" should be sent using the POST connection made earlier. But that is not happening. The connection gets closed after sending the First Chunk of data. Also I have not written the Close() method properly which I'm not sure how to implement.
I'm newbie to golang and any suggestions would be greatly helpful regarding the same.
A *strings.Reader returns io.EOF after the initial string has been read and your wrapper does nothing to change that, so it cannot be reused. You're looking for io.Pipe to turn the request body into an io.Writer.
package main
import (
"io"
"net/http"
)
func main() {
pr, pw := io.Pipe()
req, err := http.NewRequest("POST", "http://localhost:8080/upload", pr)
if err != nil {
// TODO: handle error
}
go func() {
defer pw.Close()
if _, err := io.WriteString(pw, "first chunk"); err != nil {
_ = err // TODO: handle error
}
if _, err := io.WriteString(pw, "second chunk"); err != nil {
_ = err // TODO: handle error
}
}()
res, err := http.DefaultClient.Do(req)
if err != nil {
// TODO: handle error
}
res.Body.Close()
}
Also, don't initialize the request using a struct literal. Use one of the constructors instead. In your code you're not setting the Host and Header fields, for instance.

How to listen to firestore through RPC

I want to listen to real time changes in firestore and I am also only allowed to use Go. Since firestore SDK for Go doesn't have any option to listen for real time changes, I decided to use the firestore v1beta1 sdk.
I have written the following code to do that
func TestRPCHandler(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
c, err := firestore.NewClient(context.Background())
databaseName := "projects/[project_name]/databases/(default)"
if err != nil {
panic(err)
}
stream, err := client.Listen(context.Background())
if err != nil {
panic(err)
}
request := &firestorepb.ListenRequest{
Database: databaseName,
TargetChange: &firestorepb.ListenRequest_AddTarget{
AddTarget: &firestorepb.Target{
TargetType: &firestorepb.Target_Documents{
Documents: &firestorepb.Target_DocumentsTarget{
Documents: []string{"projects/[project_name]/databases/(default)/[collection_name]"} ,
},
},
},
},
}
if err := stream.Send(request); err != nil {
panic(err)
}
if err := stream.CloseSend(); err != nil {
panic(err)
}
for {
resp, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
}
}
When I am doing this, the code does not detect any changes I bring about manually in the database. stream.Recv() just returns EOF and exits immediately. I even tried manually waiting by adding time.Sleep() but that does not help either.
You don't need the beta SDK or hacks to make this happen, I found the solution, it's pretty easy actually.
The https://firebase.google.com/docs/firestore/query-data/listen documentation does not contain an example for Go.
The source code of the firestore client API for Go has an unexported watchStream which we cannot directly use: https://github.com/googleapis/google-cloud-go/blob/master/firestore/watch.go#L130
Deep search of the repository shows that this is actually used on the DocumentSnapshotIterator and QuerySnapshotIterator at: https://github.com/googleapis/google-cloud-go/blob/master/firestore/docref.go#L644 and: https://github.com/googleapis/google-cloud-go/blob/master/firestore/query.go#L716.
The Collection contains a Snapshots method which returns the snapshot iterator that we want, after that all is easy, we just make an infivitive loop through its Next method.
Example:
cols, err := client.Collections(context.Background()).GetAll()
for _, col := range cols {
iter := col.Snapshots(context.Background())
defer iter.Stop()
for {
doc, err := iter.Next()
if err != nil {
if err == iterator.Done {
break
}
return err
}
for _, change := range doc.Changes {
// access the change.Doc returns the Document,
// which contains Data() and DataTo(&p) methods.
switch change.Kind {
case firestore.DocumentAdded:
// on added it returns the existing ones.
isNew := change.Doc.CreateTime.After(l.startTime)
// [...]
case firestore.DocumentModified:
// [...]
case firestore.DocumentRemoved:
// [...]
}
}
}
}
Yours, Gerasimos Maropoulos aka #kataras
Firebase's Get realtime updates with Cloud Firestore documentation currently indicates that Go is not yet supported.
// Not yet supported in Go client library

Golang Struct as Payload for POST Request

New to golang. I'm trying to make a POST request to an auth endpoint to get back a token for authing further requests. Currently the error I'm getting is missing "credentials". I've written the same logic in Python so I know what I am trying to do is what the system is expecting.
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/cookiejar"
"os"
)
type Auth struct {
Method string `json:"credentials"`
Email string `json:"email"`
Password string `json:"password"`
Mfa string `json:"mfa_token"`
}
func main() {
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter Email: ")
e, _ := reader.ReadString('\n')
fmt.Print("Enter Password: ")
p, _ := reader.ReadString('\n')
fmt.Print("Enter 2FA Token: ")
authy, _ := reader.ReadString('\n')
auth := Auth{"manual", e, p, authy}
j, _ := json.Marshal(auth)
jar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: jar,
}
req, err := http.NewRequest("POST", "https://internaltool.com/v3/sessions", bytes.NewBuffer(j))
if err != nil {
log.Fatal(err)
}
req.Header.Add("Accept-Encoding", "gzip, deflate, br")
res, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)
s := string(body)
if res.StatusCode == 400 {
fmt.Println("Bad Credentials")
fmt.Println(s)
return
}
}
The question is - am I properly marshalling the AUTH struct into JSON and adding it appropriately to the POST request? As the API is not even seeing the credentials key in the JSON I think I must be doing something wrong. Anything helps.
Here's a minimum viable example of using json.Marshal to convert a Struct to a JSON object in the context of a POST request.
Go's standard libraries are fantastic, there is no need to pull in external dependencies to do such a mundane thing.
func TestPostRequest(t *testing.T) {
// Create a new instance of Person
person := Person{
Name: "Ryan Alex Martin",
Age: 27,
}
// Marshal it into JSON prior to requesting
personJSON, err := json.Marshal(person)
// Make request with marshalled JSON as the POST body
resp, err := http.Post("https://httpbin.org/anything", "application/json",
bytes.NewBuffer(personJSON))
if err != nil {
t.Error("Could not make POST request to httpbin")
}
// That's it!
// But for good measure, let's look at the response body.
body, err := ioutil.ReadAll(resp.Body)
var result PersonResponse
err = json.Unmarshal([]byte(body), &result)
if err != nil {
t.Error("Error unmarshaling data from request.")
}
if result.NestedPerson.Name != "Ryan Alex Martin" {
t.Error("Incorrect or nil name field returned from server: ", result.NestedPerson.Name)
}
fmt.Println("Response from server:", result.NestedPerson.Name)
fmt.Println("Response from server:", result.NestedPerson.Age)
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
// NestedPerson is the 'json' field of the response, what we originally sent to httpbin
type PersonResponse struct {
NestedPerson Person `json:"json"` // Nested Person{} in 'json' field
}
As http.Client is relatively a low-level abstraction, gorequest(https://github.com/parnurzeal/gorequest) as an alternative is strongly recommended.
headers, queries, and body can be posted in any type, which is a bit more like what we often do in Python.

Write pipe reading into http response in golang

Here is the schema :
Client sends a POST request to server A
server A process this and sends a GET to server B
server B sends a response through A to the client
I though the best idea was to make a pipe which would read the response of the GET, and write into the response of the POST, but I got many types problems.
func main() {
r := mux.NewRouter()
r.HandleFunc("/test/{hash}", testHandler)
log.Fatal(http.ListenAndServe(":9095", r))
}
func handleErr(err error) {
if err != nil {
log.Fatalf("%s\n", err)
}
}
func testHandler(w http.ResponseWriter, r *http.Request){
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
write, _ = http.Get("http://localhost:9090/test/" + hash)
defer write.Close()
}()
w.Write(read)
}
When I run this I get the following error:
./ReverseProxy.go:61: cannot use read (type *io.PipeReader) as type []byte in argument to w.Write
Is there a way, to properly insert a io.PipeReader format into an http response?
Or am I doing this in a totally wrong way?
You are not actually writing to it, you're replacing the pipe's write.
Something along the lines of:
func testHandler(w http.ResponseWriter, r *http.Request) {
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
defer write.Close()
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
return
}
defer resp.Body.Close()
io.Copy(write, resp.Body)
}()
io.Copy(w, read)
}
Although, I agree with #JimB, for this instance, the pipe isn't even needed, something like this should be more efficient:
func testHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
hash := vars["hash"]
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
// handle error
return
}
defer resp.Body.Close()
io.Copy(w, resp.Body)
}

Access HTTP response as string in Go

I'd like to parse the response of a web request, but I'm getting trouble accessing it as string.
func main() {
resp, err := http.Get("http://google.hu/")
if err != nil {
// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
ioutil.WriteFile("dump", body, 0600)
for i:= 0; i < len(body); i++ {
fmt.Println( body[i] ) // This logs uint8 and prints numbers
}
fmt.Println( reflect.TypeOf(body) )
fmt.Println("done")
}
How can I access the response as string? ioutil.WriteFile writes correctly the response to a file.
I've already checked the package reference but it's not really helpful.
bs := string(body) should be enough to give you a string.
From there, you can use it as a regular string.
A bit as in this thread
(updated after Go 1.16 -- Q1 2021 -- ioutil deprecation: ioutil.ReadAll() => io.ReadAll()):
var client http.Client
resp, err := client.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
bodyString := string(bodyBytes)
log.Info(bodyString)
}
See also GoByExample.
As commented below (and in zzn's answer), this is a conversion (see spec).
See "How expensive is []byte(string)?" (reverse problem, but the same conclusion apply) where zzzz mentioned:
Some conversions are the same as a cast, like uint(myIntvar), which just reinterprets the bits in place.
Sonia adds:
Making a string out of a byte slice, definitely involves allocating the string on the heap. The immutability property forces this.
Sometimes you can optimize by doing as much work as possible with []byte and then creating a string at the end. The bytes.Buffer type is often useful.
The method you're using to read the http body response returns a byte slice:
func ReadAll(r io.Reader) ([]byte, error)
official documentation
You can convert []byte to a string by using
body, err := ioutil.ReadAll(resp.Body)
bodyString := string(body)
Go 1.16+ update (February 2021)
Deprecation of io/ioutil
code should be
var client http.Client
resp, err := client.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
bodyBytes, err := io.ReadAll(resp.Body)
// if u want to read the body many time
// u need to restore
// reader := io.NopCloser(bytes.NewReader(bodyBytes))
if err != nil {
log.Fatal(err)
}
bodyString := string(bodyBytes)
log.Info(bodyString)
}
reference
https://golang.org/doc/go1.16#ioutil
https://stackoverflow.com/a/52076748/2876087
string(byteslice) will convert byte slice to string, just know that it's not only simply type conversion, but also memory copy.

Resources