writing simple golang web handler.
When a request comes in, just returns if as it is.
But there are some problems.
Here is my code.
func Handler(w http.ResponseWriter, r *http.Request) {
var b []byte
var err error
if b, err = ioutil.ReadAll(r.Body); err != nil {
fmt.Println(err)
}
// r.Body is just string
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
fmt.Println(string(b)) // check r.Body string here --- (1)
// And send back as it is
if err = json.NewEncoder(w).Encode(string(b)); err != nil {
fmt.Println(err)
}
}
func TestHandler(t *testing.T) {
testStr := []rune{}
for i := 0; i < 3; i++ {
for j := 0; j < 1; j++ {
testStr = append(testStr, 'a'+rune(i))
}
testStr = append(testStr, '\n')
}
// testStr is string with '\n'
// a
// b
// c
reader := strings.NewReader(string(testStr))
req, _ := http.NewRequest("POST", "/handler", reader)
req.Header.Set("Content-Type", "text/plain")
w := httptest.NewRecorder()
Handler(w, req)
bodyBytes, _ := ioutil.ReadAll(w.Body)
fmt.Println(string(bodyBytes)) // check w.Body string here --- (2)
}
When test, the result is
(1) result
a
b
c
(2) result
"a\nb\nc\n"
How to send back as it is???
without double quotation
with newline (not string "\n")
Related
I'm trying to append an id (and other info) to the url, so I can access it later, but I can't find the right method after some research.
I've tried to use Get() method, query(), Add(), but I couldn't redirect the URL.
var email_ployer string
func RegisterNewPloyer(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/ployer/register" {
http.Error(w, "404 not found.", http.StatusNotFound)
return
}
db := connect.ConnectDB()
defer db.Close()
switch r.Method {
case "POST":
email_ployer = r.FormValue("email")
senha := r.FormValue("senha")
senha, _ = HashPassword(senha)
tx, _ := db.Begin()
stmt, _ := tx.Prepare("INSERT INTO ployers(email_ployer, senha_ployer) VALUES(?,?)")
_, erro := stmt.Exec(email_ployer, senha)
if erro != nil {
tx.Rollback()
log.Fatal(erro)
}
tx.Commit()
}
Redirect(w, r)
}
func Redirect(w http.ResponseWriter, r *http.Request) {
db2 := connect.ConnectDB()
defer db2.Close()
var id string
tx, _ := db2.Begin()
rows, _ := tx.Query("SELECT id FROM ployers WHERE email_ployer = '?'", email_ployer)
for rows.Next() {
if err := rows.Scan(&id); err != nil {
log.Fatal(err)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
}
http.Redirect(w, r, x, http.StatusSeeOther)
}
func main() {
http.HandleFunc("/ployer/seja-um-de-nos", LoadPloyerContent)
http.HandleFunc("/ployer/register", register.RegisterNewPloyer)
http.HandleFunc("/ployer/complete/", LoadPloyerContent)
http.HandleFunc("/ployer/register-received", LoadPloyerContent)
log.Fatal(http.ListenAndServe(":3306", nil))
}
In my system, I want the user to register his E-mail and password, create an new user in the DB and redirect the URL to something like localhost:3306/ployer/complete/id
Just use res.LastInsertId() to get the id and pass it to your redirect function, and build the url:
func RegisterNewPloyer(w http.ResponseWriter, r *http.Request) {
//...
//...
res, erro := stmt.Exec(email_ployer, senha)
if erro != nil {
tx.Rollback()
log.Fatal(erro)
}
tx.Commit()
id, erro := res.LastInsertId()
if erro != nil {
log.Fatal(erro)
}
Redirect(w, r, id)
}
func Redirect(w http.ResponseWriter, r *http.Request, id int64) {
uri := fmt.Sprintf("http://localhost:3306/ployer/complete/%d", id)
http.Redirect(w, r, uri, http.StatusSeeOther)
}
You should use url.Values to construct a map of query string values. Later on, you can use Encode method to generate encoded query string. Assign this value to Url.RawQuery to get the final output.
e.g.:
base, _ := url.Parse("https://www.foo.com/")
// Create an instance of Values to hold your query string parameters
values := url.Values{}
values.Add("abc", "def")
// Set the encoded output to RawQuery attribute of Url
base.RawQuery = values.Encode()
println(base.String())
If you are starting with a URL that already has query string parameters and you want add/modify items in it, use Query() method to obtain a reference to parsed map.
e.g.:
base, _ := url.Parse("https://www.foo.com/?a=b")
values := base.Query()
values.Set("a", "c")
values.Set("x", "y")
// Set the encoded output to RawQuery attribute of Url
base.RawQuery = values.Encode()
println(base.String())
I couldn't find anything helpful online on this one.
I am writing an REST API, and I want to log the size of the body of the request in bytes for metrics. Go net/http API does not provide that directly. http.Request does have Content-Length field, but that field can be empty or the client might send false data.
Is there a way to get that in the middlware level? The bruteforce method would be to read the full body and check the size. But if I do that in the middleware, the handler will not have access to the body because it would have been read and closed.
Why do you want a middle in here?
The simple way is b, err = io.Copy(anyWriterOrMultiwriter, r.Body)
b is total content length of request when err == nil
Use request body as you want. Also b, err = io.Copy(ioutil.Discard, r.Body)
You could write a custom ReadCloser that proxies an existing one and counts bytes as it goes. Something like:
type LengthReader struct {
Source io.ReadCloser
Length int
}
func (r *LengthReader) Read(b []byte) (int, error) {
n, err := r.Source.Read(b)
r.Length += n
return n, err
}
func (r *LengthReader) Close() error {
var buf [32]byte
var n int
var err error
for err == nil {
n, err = r.Source.Read(buf[:])
r.Length += n
}
closeerr := r.Source.Close()
if err != nil && err != io.EOF {
return err
}
return closeerr
}
This will count bytes as you read them from the stream, and when closed it will consume and count all remaining unread bytes first. After you're finished with the stream, you can then access the length.
Option 1
Use TeeReader and this is scalable. It splits reader into two and one of them calculates the size using allocated memory. Also, in the first case
maxmem := 4096
var buf bytes.Buffer
// comment this line out if you want to disable gathering metrics
resp.Body = io.TeeReader(resp.Body, &buf)
readsize := func(r io.Reader) int {
bytes := make([]byte, maxmem)
var size int
for {
read, err := r.Read(bytes)
if err == io.EOF {
break
}
size += read
}
return size
}
log.Printf("Size is %d", readsize(&buf))
Option 2 unscalable way (original answer)
You can just read the body, calculate the size, then unmarshal into struct, so that it becomes:
b, _ := ioutil.ReadAll(r.Body)
size := len(b) // can be nil so check err in your app
if err := json.Unmarshal(b, &input); err != nil {
s.BadReq(w, errors.New("error reading body"))
return
}
My code:
func getSourceUrl(url string) (string, error) {
resp, err := http.Get(url)
if err != nil {
fmt.Println("Error getSourceUrl: ")
return "", err
}
defer resp.Body.Close()
body := resp.Body
// time = 0
sourcePage, err := ioutil.ReadAll(body)
// time > 5 minutes
return string(sourcePage), err
}
I have a website link with a source of around> 100000 lines. Using ioutil.ReadAll made me get very long (about> 5 minutes for 1 link). Is there a way to get Source website faster? Thank you!
#Minato try this code, play with M throttling parameter. Play with it if you get too errors (reduce it).
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"runtime"
"time"
)
// Token is an empty struct for signalling
type Token struct{}
// N files to get
var N = 301 // at the source 00000 - 00300
// M max go routines
var M = runtime.NumCPU() * 16
// Throttle to max M go routines
var Throttle = make(chan Token, M)
// DoneStatus is used to signal end of
type DoneStatus struct {
length int
sequence string
duration float64
err error
}
// ExitOK is simple exit counter
var ExitOK = make(chan DoneStatus)
// TotalBytes read
var TotalBytes = 0
// TotalErrors captured
var TotalErrors = 0
// URLTempl is templte for URL construction
var URLTempl = "https://virusshare.com/hashes/VirusShare_%05d.md5"
func close(c io.Closer) {
err := c.Close()
if err != nil {
log.Fatal(err)
}
}
func main() {
log.Printf("start main. M=%d\n", M)
startTime := time.Now()
for i := 0; i < N; i++ {
go func(idx int) {
// slow ramp up fire getData after i seconds
time.Sleep(time.Duration(i) * time.Second)
url := fmt.Sprintf(URLTempl, idx)
_, _ = getData(url) // errors captured as data
}(i)
}
// Count N byte count signals
for i := 0; i < N; i++ {
status := <-ExitOK
TotalBytes += status.length
if status.err != nil {
TotalErrors++
log.Printf("[%d] : %v\n", i, status.err)
continue
}
log.Printf("[%d] file %s, %.1f MByte, %.1f min, %.1f KByte/sec\n",
i, status.sequence,
float64(status.length)/(1024*1024),
status.duration/60,
float64(status.length)/(1024)/status.duration)
}
// totals
duration := time.Since(startTime).Seconds()
log.Printf("Totals: %.1f MByte, %.1f min, %.1f KByte/sec\n",
float64(TotalBytes)/(1024*1024),
duration/60,
float64(TotalBytes)/(1024)/duration)
// using fatal to verify only one go routine is running at the end
log.Fatalf("TotalErrors: %d\n", TotalErrors)
}
func getData(url string) (data []byte, err error) {
var startTime time.Time
defer func() {
// release token
<-Throttle
// signal end of go routine, with some status info
ExitOK <- DoneStatus{
len(data),
url[41:46],
time.Since(startTime).Seconds(),
err,
}
}()
// acquire one of M tokens
Throttle <- Token{}
log.Printf("Started file: %s\n", url[41:46])
startTime = time.Now()
resp, err := http.Get(url)
if err != nil {
return
}
defer close(resp.Body)
data, err = ioutil.ReadAll(resp.Body)
if err != nil {
return
}
return
}
Per transfer variation is about 10-40KByte/sec and final total for all 301 files I get 928MB, 11.1min at 1425 KByte/sec. I believe you should be able to get similar results.
// outside the scope of the question but maybe useful
Also give this a try http://www.dslreports.com/speedtest/ go to settings and select bunch of US servers for testing and set duration to 60sec. This will tell you what your actual effective total rate is to US.
Good luck!
You could iterate sections of the response at a time, something like;
responseSection := make([]byte, 128)
body.Read(responseSection)
return string(responseSection), err
Which would read 128 bytes at a time. However would suggest confirming the download speed is not causing the slow load.
The 5 minutes is probably network time.
That said, you generally would not want to buffer enormous objects in memory.
resp.Body is a Reader.
So you cold use io.Copy to copy its contents into a file.
Converting sourcePage into a string is a bad idea as it forces another allocation.
I want to call FormValue on golang net/http Request from several middleware handler functions before serving the request. And I do not want to invalidate the request while doing this.
It works fine except when the incoming request has a multipart form-data, the data gets invalidated after I call FormValue and there is nothing to parse in the final route.
I wrote a utility function that solved my problem:
package utils
import (
"bytes"
"io"
"io/ioutil"
"mime"
"mime/multipart"
"net/http"
"strings"
)
// Get form values without invalidating the request body in case the data is multiform
func GetFormValues(request *http.Request, keys []string) []string {
var values []string
mediaType, params, err := mime.ParseMediaType(request.Header.Get("Content-Type"))
if err != nil || !strings.HasPrefix(mediaType, "multipart/") {
for i := range keys {
values = append(values, request.FormValue(keys[i]))
}
} else { // multi form
buf, _ := ioutil.ReadAll(request.Body)
origBody := ioutil.NopCloser(bytes.NewBuffer(buf))
var rdr = multipart.NewReader(bytes.NewBuffer(buf), params["boundary"])
for len(values) < len(keys) {
part, err_part := rdr.NextPart()
if err_part == io.EOF {
break
}
for i := range keys {
if part.FormName() == keys[i] {
buf := new(bytes.Buffer)
buf.ReadFrom(part)
values = append(values, buf.String())
}
}
}
request.Body = origBody
}
if len(values) == len(keys) {
return values
} else {
return nil
}
}
// Get form value without invalidating the request body in case the data is multiform
func GetFormValue(request *http.Request, key string) string {
if result := GetFormValues(request, []string{key}); len(result) == 1 {
return result[0]
} else {
return ""
}
}
Now instead of calling
value := request.FormValue(key)
I do
value := utils.GetFormValue(request, key)
or for multiple values
values := utils.GetFormValues(request, []string{keys...})
I have a client server application, using TCP connection
Client:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
//read M and N
...
tcpAddr, err := net.ResolveTCPAddr("tcp4", service)
...
var p P
p.M = M
p.N = N
err = enc.Encode(p)
}
Server:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
tcpAddr, err := net.ResolveTCPAddr("ip4", service)
listener, err := net.ListenTCP("tcp", tcpAddr)
...
var connB bytes.Buffer
dec := gob.NewDecoder(&connB)
var p P
err = dec.Decode(p)
fmt.Printf("{%d, %d}\n", p.M, p.N)
}
The result on serve is {0, 0} because I don't know how to obtain a bytes.Buffer variable from net.Conn.
Is there any way for sending gob variables over TCP ?
If true, how can this be done ? Or there are any alternative in sending numbers over TCP ?
Any help or sample code would really be appreciated.
Here's a complete example.
Server:
package main
import (
"fmt"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func handleConnection(conn net.Conn) {
dec := gob.NewDecoder(conn)
p := &P{}
dec.Decode(p)
fmt.Printf("Received : %+v", p);
conn.Close()
}
func main() {
fmt.Println("start");
ln, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
for {
conn, err := ln.Accept() // this blocks until connection or error
if err != nil {
// handle error
continue
}
go handleConnection(conn) // a goroutine handles conn so that the loop can accept other connections
}
}
Client :
package main
import (
"fmt"
"log"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func main() {
fmt.Println("start client");
conn, err := net.Dial("tcp", "localhost:8080")
if err != nil {
log.Fatal("Connection error", err)
}
encoder := gob.NewEncoder(conn)
p := &P{1, 2}
encoder.Encode(p)
conn.Close()
fmt.Println("done");
}
Launch the server, then the client, and you see the server displaying the received P value.
A few observations to make it clear :
When you listen on a socket, you should pass the open socket to a goroutine that will handle it.
Conn implements the Reader and Writer interfaces, which makes it easy to use : you can give it to a Decoder or Encoder
In a real application you would probably have the P struct definition in a package imported by both programs