serving content of unbounded size in http in golang - http

I have the following server code:
package main
import (
"fmt"
"net/http"
"time"
)
type serveData struct {
}
func (s *serveData) Read (p []byte) (int, error) {
l := len (p)
fmt.Println ("p size is ", l);
time.Sleep(200 * time.Millisecond);
return l, nil;
}
func (s *serveData) Seek (offset int64, whence int) (int64, error) {
fmt.Println ("in seek ", offset);
return offset, nil;
}
func handler(w http.ResponseWriter, r *http.Request) {
reader := new (serveData)
//w.WriteHeader(206);
w.Header().Set("Content-type", "application/octet-stream");
fmt.Println ("got request");
http.ServeContent(w, r, "cool", time.Date(2009, time.November, 10, 23, 0, 0, 0, time.UTC), reader)
}
func main() {
http.HandleFunc("/check", handler)
http.ListenAndServe(":8080", nil)
}
When i connect the client to the server above (curl -X GET http://127.0.0.1:8080/check), nothing happens and curl just exits. The server calls Seek() function 2 times, but never calls Read()
I am serving partial content, as the size is unbounded (pseudo-live data). Also, when I uncomment w.WriteHeader(206), the server complains about "http: multiple response.WriteHeader calls"
What possibly is going wrong here?

Related

golang http server handler and pointer issue

I am a bit lost with go-routines, http server, handlers and pointers.
I ve created a http server that accepts any request at any path, I am starting it in separate routine so main thread is not blocked
// main thread
// I want to bind to random port and read port later
listener, err := net.Listen("tcp", ":0")
rh := requestHandler{} // look at point 2)
s := &http.Server{
Handler: rh,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
MaxHeaderBytes: 1 << 20,
}
go func() {
err := s.Serve(listener)
...
}()
rh is a Handler interface so I ve created a struct following the interface
type requestHandler struct {
values []string
}
func (rh requestHandler) ServeHTTP(w http.ResponseWriter, givenRequest *http.Request) {
log.Println("DEBUG : handling request", givenRequest.Method, givenRequest.URL, givenRequest.RequestURI)
log.Println("DEBUG : headers", givenRequest.Header)
for _, value := range rh.values { // <----- values are always empty
}
}
lets add some values to handler
// main thread
rh.values = append(rh.values, "one)
obviously this doesn't work as Server is copying Handler and rh variable from main thread has different pointer than s.Handler, but this is OK.
What I dont understand is that also rh in func (rh requestHandler) ServeHTTP(w http.ResponseWriter, givenRequest *http.Request) has different pointer than s.Handler.
how can I pass anything to requestHandler.values from main thread so those values can be read in callback function ? I ve tried using channels but I cannot access pointer/ref to right rh. Is it because each request is also spawn in new routine and handler is also copied ?
sample code
type MyServer struct {
Port int
pid *http.Server
requestHandler requestHandler
}
func (s *MyServer) addValue(value string) {
s.requestHandler.values = append(s.requestHandler.values, value)
}
func InitializeServer() *MyServer {
rh := requestHandler{}
listener := bindToNextFreePort()
s := &http.Server{
Handler: rh,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
MaxHeaderBytes: 1 << 20,
}
startServer(s, listener)
currentPort := getTcpPort(listener)
log.Println("My Server Initialized on ", currentPort)
return &MyServer{
pid: s,
Port: currentPort,
requestHandler: rh,
}
}
func startServer(s *http.Server, listener net.Listener) {
go func() {
log.Println("starting Mock Server")
err := s.Serve(listener)
}()
}
type requestHandler struct {
values []string
}
func (rh requestHandler) ServeHTTP(w http.ResponseWriter, givenRequest *http.Request) {
log.Println("DEBUG : handling request", givenRequest.Method, givenRequest.URL, givenRequest.RequestURI)
log.Println("DEBUG : headers", givenRequest.Header)
for key, value := range rh.values {
}
}
func bindToNextFreePort() net.Listener {
listener, err := net.Listen("tcp", ":0")
if err != nil {
panic(err)
}
return listener
}
It turned out to be quite simple, I've closed my eyes for this assignment here
return &MyServer{
pid: s,
Port: currentPort,
requestHandler: rh,
}
Which led to yet another copy.

How can I set up the logging context from middleware?

I want to populate the logging context by items in the request, for example: r.Header.Get("X-Request-Id"). I assumed I could override the Log type in the handler from middleware. Though it doesn't seem to work and I am not sure why!
package main
import (
"fmt"
"net/http"
"os"
"github.com/apex/log"
"github.com/gorilla/mux"
)
// Assumption: handler is the shared state between the functions
type handler struct{ Log *log.Entry }
// New creates a handler for this application to co-ordinate shared resources
func New() (h handler) { return handler{Log: log.WithFields(log.Fields{"test": "FAIL"})} }
func (h handler) index(w http.ResponseWriter, r *http.Request) {
h.Log.Info("Hello from the logger")
fmt.Fprint(w, "YO")
}
func main() {
h := New()
app := mux.NewRouter()
app.HandleFunc("/", h.index)
app.Use(h.loggingMiddleware)
if err := http.ListenAndServe(":"+os.Getenv("PORT"), app); err != nil {
log.WithError(err).Fatal("error listening")
}
}
func (h handler) loggingMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h.Log = log.WithFields(log.Fields{"test": "PASS"})
next.ServeHTTP(w, r)
})
}
Can you see why h.Log = log.WithFields(log.Fields{"test": "PASS"}) doesn't seem to have any effect on h.Log.Info("Hello from the logger") which should be IIUC within the same request?
You need your logger to be request-scoped. You're setting it globally for the entire handler, every time a new connection comes in, which means you're asking for data races, and generally undesirable behavior.
For request-scoped context, the context.Context embedded in the request is perfect. You can access it through the Context() and WithContext methods.
Example:
var loggerKey = "Some unique key"
func (h handler) loggingMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
ctx = context.WithValue(ctx, loggerKey, log.WithFields(log.Fields{"test": "PASS"}))
next.ServeHTTP(w, r.WithContext(ctx)
})
}
Then to access your logger:
func doSomething(r *http.Request) error {
log, ok := r.Context().Value(loggerKey).(*log.Logger) // Or whatever type is appropriate
if !ok {
return errors.New("logger not set on context!")
}
// Do stuff...
}

How to retrieve website source without using ioutil.ReadAll in golang

My code:
func getSourceUrl(url string) (string, error) {
resp, err := http.Get(url)
if err != nil {
fmt.Println("Error getSourceUrl: ")
return "", err
}
defer resp.Body.Close()
body := resp.Body
// time = 0
sourcePage, err := ioutil.ReadAll(body)
// time > 5 minutes
return string(sourcePage), err
}
I have a website link with a source of around> 100000 lines. Using ioutil.ReadAll made me get very long (about> 5 minutes for 1 link). Is there a way to get Source website faster? Thank you!
#Minato try this code, play with M throttling parameter. Play with it if you get too errors (reduce it).
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"runtime"
"time"
)
// Token is an empty struct for signalling
type Token struct{}
// N files to get
var N = 301 // at the source 00000 - 00300
// M max go routines
var M = runtime.NumCPU() * 16
// Throttle to max M go routines
var Throttle = make(chan Token, M)
// DoneStatus is used to signal end of
type DoneStatus struct {
length int
sequence string
duration float64
err error
}
// ExitOK is simple exit counter
var ExitOK = make(chan DoneStatus)
// TotalBytes read
var TotalBytes = 0
// TotalErrors captured
var TotalErrors = 0
// URLTempl is templte for URL construction
var URLTempl = "https://virusshare.com/hashes/VirusShare_%05d.md5"
func close(c io.Closer) {
err := c.Close()
if err != nil {
log.Fatal(err)
}
}
func main() {
log.Printf("start main. M=%d\n", M)
startTime := time.Now()
for i := 0; i < N; i++ {
go func(idx int) {
// slow ramp up fire getData after i seconds
time.Sleep(time.Duration(i) * time.Second)
url := fmt.Sprintf(URLTempl, idx)
_, _ = getData(url) // errors captured as data
}(i)
}
// Count N byte count signals
for i := 0; i < N; i++ {
status := <-ExitOK
TotalBytes += status.length
if status.err != nil {
TotalErrors++
log.Printf("[%d] : %v\n", i, status.err)
continue
}
log.Printf("[%d] file %s, %.1f MByte, %.1f min, %.1f KByte/sec\n",
i, status.sequence,
float64(status.length)/(1024*1024),
status.duration/60,
float64(status.length)/(1024)/status.duration)
}
// totals
duration := time.Since(startTime).Seconds()
log.Printf("Totals: %.1f MByte, %.1f min, %.1f KByte/sec\n",
float64(TotalBytes)/(1024*1024),
duration/60,
float64(TotalBytes)/(1024)/duration)
// using fatal to verify only one go routine is running at the end
log.Fatalf("TotalErrors: %d\n", TotalErrors)
}
func getData(url string) (data []byte, err error) {
var startTime time.Time
defer func() {
// release token
<-Throttle
// signal end of go routine, with some status info
ExitOK <- DoneStatus{
len(data),
url[41:46],
time.Since(startTime).Seconds(),
err,
}
}()
// acquire one of M tokens
Throttle <- Token{}
log.Printf("Started file: %s\n", url[41:46])
startTime = time.Now()
resp, err := http.Get(url)
if err != nil {
return
}
defer close(resp.Body)
data, err = ioutil.ReadAll(resp.Body)
if err != nil {
return
}
return
}
Per transfer variation is about 10-40KByte/sec and final total for all 301 files I get 928MB, 11.1min at 1425 KByte/sec. I believe you should be able to get similar results.
// outside the scope of the question but maybe useful
Also give this a try http://www.dslreports.com/speedtest/ go to settings and select bunch of US servers for testing and set duration to 60sec. This will tell you what your actual effective total rate is to US.
Good luck!
You could iterate sections of the response at a time, something like;
responseSection := make([]byte, 128)
body.Read(responseSection)
return string(responseSection), err
Which would read 128 bytes at a time. However would suggest confirming the download speed is not causing the slow load.
The 5 minutes is probably network time.
That said, you generally would not want to buffer enormous objects in memory.
resp.Body is a Reader.
So you cold use io.Copy to copy its contents into a file.
Converting sourcePage into a string is a bad idea as it forces another allocation.

What is the proper way to detect a dropped connection in Go?

I am working with a Go HTTP server implementation that reads an upload from a mobile client. However, I'm experiencing problems where because of a long keep-alive, the server will hang reading the request buffer for quite a long time if the mobile client goes offline (as often happens).
What is the proper way to detect a dropped connection and close the input buffer?
Set a reasonable timeout on the server, for example:
srv := &http.Server{
Addr: ":443",
ReadTimeout: time.Minute * 2,
WriteTimeout: time.Minute * 2,
}
log.Fatal(srv.ListenAndServeTLS(certFile, keyFile))
Because I wanted to drop the connection only when writes stop (and quickly, so that I can record the data received so far and allow the client to resume), the ReadTimeout isn't the right solution for me.
I found the answer in this gist. You need to set a read on the connection itself.
package nettimeout
import (
"net"
"time"
)
// Listener wraps a net.Listener, and gives a place to store the timeout
// parameters. On Accept, it will wrap the net.Conn with our own Conn for us.
type Listener struct {
net.Listener
ReadTimeout time.Duration
WriteTimeout time.Duration
}
func (l *Listener) Accept() (net.Conn, error) {
c, err := l.Listener.Accept()
if err != nil {
return nil, err
}
tc := &Conn{
Conn: c,
ReadTimeout: l.ReadTimeout,
WriteTimeout: l.WriteTimeout,
}
return tc, nil
}
// Conn wraps a net.Conn, and sets a deadline for every read
// and write operation.
type Conn struct {
net.Conn
ReadTimeout time.Duration
WriteTimeout time.Duration
}
func (c *Conn) Read(b []byte) (int, error) {
err := c.Conn.SetReadDeadline(time.Now().Add(c.ReadTimeout))
if err != nil {
return 0, err
}
return c.Conn.Read(b)
}
func (c *Conn) Write(b []byte) (int, error) {
err := c.Conn.SetWriteDeadline(time.Now().Add(c.WriteTimeout))
if err != nil {
return 0, err
}
return c.Conn.Write(b)
}
func NewListener(addr string, readTimeout, writeTimeout time.Duration) (net.Listener, error) {
l, err := net.Listen("tcp", addr)
if err != nil {
return nil, err
}
tl := &Listener{
Listener: l,
ReadTimeout: readTimeout,
WriteTimeout: writeTimeout,
}
return tl, nil
}

Unable to send gob data over TCP in Go Programming

I have a client server application, using TCP connection
Client:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
//read M and N
...
tcpAddr, err := net.ResolveTCPAddr("tcp4", service)
...
var p P
p.M = M
p.N = N
err = enc.Encode(p)
}
Server:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
tcpAddr, err := net.ResolveTCPAddr("ip4", service)
listener, err := net.ListenTCP("tcp", tcpAddr)
...
var connB bytes.Buffer
dec := gob.NewDecoder(&connB)
var p P
err = dec.Decode(p)
fmt.Printf("{%d, %d}\n", p.M, p.N)
}
The result on serve is {0, 0} because I don't know how to obtain a bytes.Buffer variable from net.Conn.
Is there any way for sending gob variables over TCP ?
If true, how can this be done ? Or there are any alternative in sending numbers over TCP ?
Any help or sample code would really be appreciated.
Here's a complete example.
Server:
package main
import (
"fmt"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func handleConnection(conn net.Conn) {
dec := gob.NewDecoder(conn)
p := &P{}
dec.Decode(p)
fmt.Printf("Received : %+v", p);
conn.Close()
}
func main() {
fmt.Println("start");
ln, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
for {
conn, err := ln.Accept() // this blocks until connection or error
if err != nil {
// handle error
continue
}
go handleConnection(conn) // a goroutine handles conn so that the loop can accept other connections
}
}
Client :
package main
import (
"fmt"
"log"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func main() {
fmt.Println("start client");
conn, err := net.Dial("tcp", "localhost:8080")
if err != nil {
log.Fatal("Connection error", err)
}
encoder := gob.NewEncoder(conn)
p := &P{1, 2}
encoder.Encode(p)
conn.Close()
fmt.Println("done");
}
Launch the server, then the client, and you see the server displaying the received P value.
A few observations to make it clear :
When you listen on a socket, you should pass the open socket to a goroutine that will handle it.
Conn implements the Reader and Writer interfaces, which makes it easy to use : you can give it to a Decoder or Encoder
In a real application you would probably have the P struct definition in a package imported by both programs

Resources