I'm coding a ShareX clone for Linux in Go that uploads files and images to file sharing services through http POST requests.
I'm currently using http.Client and Do() to send my requests, but I'd like to be able to track the upload progress for bigger files that take up to a minute to upload.
The only way I can think of at the moment is manually opening a TCP connection on port 80 to the website and write the HTTP request in chunks, but I don't know if it would work on https sites and I'm not sure if it's the best way to do it.
Is there any other way to achieve this?
You can create your own io.Reader to wrap the actual reader and then you can output the progress each time Read is called.
Something along the lines of:
type ProgressReader struct {
io.Reader
Reporter func(r int64)
}
func (pr *ProgressReader) Read(p []byte) (n int, err error) {
n, err = pr.Reader.Read(p)
pr.Reporter(int64(n))
return
}
func main() {
file, _ := os.Open("/tmp/blah.go")
total := int64(0)
pr := &ProgressReader{file, func(r int64) {
total += r
if r > 0 {
fmt.Println("progress", r)
} else {
fmt.Println("done", r)
}
}}
io.Copy(ioutil.Discard, pr)
}
Wrap the reader passed as the request body with something that reports progress. For example,
type progressReporter struct {
r io.Reader
max int
sent int
}
func (pr *progressReader) Read(p []byte) (int, error) {
n, err := pr.r.Read(p)
pr.sent += n
if err == io.EOF {
pr.atEOF = true
}
pr.report()
return n, err
}
func (pr *progressReporter) report() {
fmt.Printf("sent %d of %d bytes\n", pr.sent, pr.max)
if pr.atEOF {
fmt.Println("DONE")
}
}
If previously you called
client.Post(u, contentType, r)
then change the code to
client.Post(u, contentType, &progressReader{r:r, max:max})
where max is the number of bytes you expect to send. Modify the progressReporter.report() method and add fields to progressReporter to meet your specific needs.
Related
I couldn't find anything helpful online on this one.
I am writing an REST API, and I want to log the size of the body of the request in bytes for metrics. Go net/http API does not provide that directly. http.Request does have Content-Length field, but that field can be empty or the client might send false data.
Is there a way to get that in the middlware level? The bruteforce method would be to read the full body and check the size. But if I do that in the middleware, the handler will not have access to the body because it would have been read and closed.
Why do you want a middle in here?
The simple way is b, err = io.Copy(anyWriterOrMultiwriter, r.Body)
b is total content length of request when err == nil
Use request body as you want. Also b, err = io.Copy(ioutil.Discard, r.Body)
You could write a custom ReadCloser that proxies an existing one and counts bytes as it goes. Something like:
type LengthReader struct {
Source io.ReadCloser
Length int
}
func (r *LengthReader) Read(b []byte) (int, error) {
n, err := r.Source.Read(b)
r.Length += n
return n, err
}
func (r *LengthReader) Close() error {
var buf [32]byte
var n int
var err error
for err == nil {
n, err = r.Source.Read(buf[:])
r.Length += n
}
closeerr := r.Source.Close()
if err != nil && err != io.EOF {
return err
}
return closeerr
}
This will count bytes as you read them from the stream, and when closed it will consume and count all remaining unread bytes first. After you're finished with the stream, you can then access the length.
Option 1
Use TeeReader and this is scalable. It splits reader into two and one of them calculates the size using allocated memory. Also, in the first case
maxmem := 4096
var buf bytes.Buffer
// comment this line out if you want to disable gathering metrics
resp.Body = io.TeeReader(resp.Body, &buf)
readsize := func(r io.Reader) int {
bytes := make([]byte, maxmem)
var size int
for {
read, err := r.Read(bytes)
if err == io.EOF {
break
}
size += read
}
return size
}
log.Printf("Size is %d", readsize(&buf))
Option 2 unscalable way (original answer)
You can just read the body, calculate the size, then unmarshal into struct, so that it becomes:
b, _ := ioutil.ReadAll(r.Body)
size := len(b) // can be nil so check err in your app
if err := json.Unmarshal(b, &input); err != nil {
s.BadReq(w, errors.New("error reading body"))
return
}
Im trying to build a small website, I use the html/template to create dynamic pages. One thing on the pages is a list of URL's inside those urls sometimes I need character encoding. for special characters like ô (%C3%B4).
When i try to parse the variables into a page using html/template i get the following as a result: %!c(MISSING)3%!b(MISSING)4. I have no clue what is wrong here
type Search_list struct {
Search_name string
Search_url string
Search_price float64
}
func generateSearchPage(language int, q string) (string, error) {
/* ommited, fetshing data from elasticsrearch*/
sl := []Search_list{}
var urle *url.URL
//looping through ES results and putting them in a custom List
for _, res := range data.Hits.Hits {
//
//Encode Url
var err error
urle, err = url.Parse(res.Source.URL)
if err != nil {
continue
// TODO: add log
}
//I've tried already the following:
fmt.Println(res.Source.URL) //ô
fmt.Println(url.QueryUnescape(res.Source.URL)) //ô
fmt.Println(urle.String()) //%C3%B4
u, _ := url.QueryUnescape(res.Source.URL)
sl = append(sl, Search_list{res.Source.Name, u, res.Source.Price})
}
var buffer bytes.Buffer
t := template.New("Index template")
t, err = t.Parse(page_layout[language][PageTypeSearch])
if err != nil {
panic(err)
}
err = t.Execute(&buffer, Search_data{
Title: translations[language]["homepage"],
Page_title: WebSiteName,
Listed_items: sl,
})
if err != nil {
panic(err)
}
return buffer.String(), nil // %!c(MISSING)3%!b(MISSING)4
}
# Moshe Revah
thanks for the help, in the meantime I found the error
Later in the code I send my generated page to the http client with
fmt.Fprintf(w, page) // Here was the error b/c of the % symbols
I just changed it to
fmt.Fprint(w, page)
and it works perfect
My code:
func getSourceUrl(url string) (string, error) {
resp, err := http.Get(url)
if err != nil {
fmt.Println("Error getSourceUrl: ")
return "", err
}
defer resp.Body.Close()
body := resp.Body
// time = 0
sourcePage, err := ioutil.ReadAll(body)
// time > 5 minutes
return string(sourcePage), err
}
I have a website link with a source of around> 100000 lines. Using ioutil.ReadAll made me get very long (about> 5 minutes for 1 link). Is there a way to get Source website faster? Thank you!
#Minato try this code, play with M throttling parameter. Play with it if you get too errors (reduce it).
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"runtime"
"time"
)
// Token is an empty struct for signalling
type Token struct{}
// N files to get
var N = 301 // at the source 00000 - 00300
// M max go routines
var M = runtime.NumCPU() * 16
// Throttle to max M go routines
var Throttle = make(chan Token, M)
// DoneStatus is used to signal end of
type DoneStatus struct {
length int
sequence string
duration float64
err error
}
// ExitOK is simple exit counter
var ExitOK = make(chan DoneStatus)
// TotalBytes read
var TotalBytes = 0
// TotalErrors captured
var TotalErrors = 0
// URLTempl is templte for URL construction
var URLTempl = "https://virusshare.com/hashes/VirusShare_%05d.md5"
func close(c io.Closer) {
err := c.Close()
if err != nil {
log.Fatal(err)
}
}
func main() {
log.Printf("start main. M=%d\n", M)
startTime := time.Now()
for i := 0; i < N; i++ {
go func(idx int) {
// slow ramp up fire getData after i seconds
time.Sleep(time.Duration(i) * time.Second)
url := fmt.Sprintf(URLTempl, idx)
_, _ = getData(url) // errors captured as data
}(i)
}
// Count N byte count signals
for i := 0; i < N; i++ {
status := <-ExitOK
TotalBytes += status.length
if status.err != nil {
TotalErrors++
log.Printf("[%d] : %v\n", i, status.err)
continue
}
log.Printf("[%d] file %s, %.1f MByte, %.1f min, %.1f KByte/sec\n",
i, status.sequence,
float64(status.length)/(1024*1024),
status.duration/60,
float64(status.length)/(1024)/status.duration)
}
// totals
duration := time.Since(startTime).Seconds()
log.Printf("Totals: %.1f MByte, %.1f min, %.1f KByte/sec\n",
float64(TotalBytes)/(1024*1024),
duration/60,
float64(TotalBytes)/(1024)/duration)
// using fatal to verify only one go routine is running at the end
log.Fatalf("TotalErrors: %d\n", TotalErrors)
}
func getData(url string) (data []byte, err error) {
var startTime time.Time
defer func() {
// release token
<-Throttle
// signal end of go routine, with some status info
ExitOK <- DoneStatus{
len(data),
url[41:46],
time.Since(startTime).Seconds(),
err,
}
}()
// acquire one of M tokens
Throttle <- Token{}
log.Printf("Started file: %s\n", url[41:46])
startTime = time.Now()
resp, err := http.Get(url)
if err != nil {
return
}
defer close(resp.Body)
data, err = ioutil.ReadAll(resp.Body)
if err != nil {
return
}
return
}
Per transfer variation is about 10-40KByte/sec and final total for all 301 files I get 928MB, 11.1min at 1425 KByte/sec. I believe you should be able to get similar results.
// outside the scope of the question but maybe useful
Also give this a try http://www.dslreports.com/speedtest/ go to settings and select bunch of US servers for testing and set duration to 60sec. This will tell you what your actual effective total rate is to US.
Good luck!
You could iterate sections of the response at a time, something like;
responseSection := make([]byte, 128)
body.Read(responseSection)
return string(responseSection), err
Which would read 128 bytes at a time. However would suggest confirming the download speed is not causing the slow load.
The 5 minutes is probably network time.
That said, you generally would not want to buffer enormous objects in memory.
resp.Body is a Reader.
So you cold use io.Copy to copy its contents into a file.
Converting sourcePage into a string is a bad idea as it forces another allocation.
I have a server and start a goroutine for sending data. Code like this
func handleSend(conn *net.TcpConn, ch <-chan []byte) {
for {
select {
case msg, ok := <-ch:
if !ok {
return
}
n, err := conn.Write(msg)
if err != nil {
log.Error("conn write error", err)
return
}
//here
}
}
}
Today, I think I should check n return by conn.Write to make sure that msg is writed completely. So I add the following code in the place here
for ;n!= len(msg);{
log.Error("conn write not completely", len(msg), "actually", n)
msg = msg[n:]
n, err = sess.conn.Write(msg)
if err != nil {
log.Error("conn write error", err)
return
}
}
And I want to know is it right to do this?
PS what if the peer receive message slowly and Write return with only part of the data sended successfully ?
Docs say Write must return a non-nil error if it returns n < len(p). You may well find there's not much you can do with n, in which case you can assign it to _.
As an exercise I created a small HTTP server that generates random game mechanics, similar to this one. I wrote it on a Windows 7 (32-bit) system and it works flawlessly. However, when I run it on my home machine, Windows 7 (64-bit), it always fails with the same message: exit status -1073741819. I haven't managed to find anything on the web which references that status code, so I don't know how important it is.
Here's code for the server, with redundancy abridged:
package main
import (
"fmt"
"math/rand"
"time"
"net/http"
"html/template"
)
// Info about a game mechanic
type MechanicInfo struct { Name, Desc string }
// Print a mechanic as a string
func (m MechanicInfo) String() string {
return fmt.Sprintf("%s: %s", m.Name, m.Desc)
}
// A possible game mechanic
var (
UnkillableObjects = &MechanicInfo{"Avoiding Unkillable Objects",
"There are objects that the player cannot touch. These are different from normal enemies because they cannot be destroyed or moved."}
//...
Race = &MechanicInfo{"Race",
"The player must reach a place before the opponent does. Like \"Timed\" except the enemy as a \"timer\" can be slowed down by the player's actions, or there may be multiple enemies being raced against."}
)
// Slice containing all game mechanics
var GameMechanics []*MechanicInfo
// Pseudorandom number generator
var prng *rand.Rand
// Get a random mechanic
func RandMechanic() *MechanicInfo {
i := prng.Intn(len(GameMechanics))
return GameMechanics[i]
}
// Initialize the package
func init() {
prng = rand.New(rand.NewSource(time.Now().Unix()))
GameMechanics = make([]*MechanicInfo, 34)
GameMechanics[0] = UnkillableObjects
//...
GameMechanics[33] = Race
}
// serving
var index = template.Must(template.ParseFiles(
"templates/_base.html",
"templates/index.html",
))
func randMechHandler(w http.ResponseWriter, req *http.Request) {
mechanics := [3]*MechanicInfo{RandMechanic(), RandMechanic(), RandMechanic()}
if err := index.Execute(w, mechanics); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
func main() {
http.HandleFunc("/", randMechHandler)
if err := http.ListenAndServe(":80", nil); err != nil {
panic(err)
}
}
In addition, the unabridged code, the _base.html template, and the index.html template.
What could be causing this issue? Is there a process for debugging a cryptic exit status like this?
When I ran it, I got the following two errors:
template: content:6: nil pointer evaluating *main.MechanicInfo.Name
http: multiple response.WriteHeader calls
The former was in the web browser, the latter in the console window where I launched your server.
The nil pointer problem is because your abridged program leaves GameMechanics[1:32] set to nil.
The second error is interesting. The only place in your program that any methods on your http.ResponseWriter get called is inside of index.Execute, which is not your code -- meaning maybe there is something wrong happening in html/template. I'm testing this with Go 1.0.2.
I put _base.html at the top of index.html and then changed index to this:
var index = template.Must(template.ParseFiles("templates/index.html"))
and the http.WriteHeaders warning went away.
Not really an answer, but a direction you could explore.
As a bonus, here's the more "Go way" of writing your program. Note that I simplified the use of the PRNG (you don't need to instantiate unless you want several going in parallel) and simplified the structure initializer:
package main
import (
"fmt"
"html/template"
"math/rand"
"net/http"
)
// Info about a game mechanic
type MechanicInfo struct{ Name, Desc string }
// Print a mechanic as a string
func (m MechanicInfo) String() string {
return fmt.Sprintf("%s: %s", m.Name, m.Desc)
}
// The game mechanics
var GameMechanics = [...]*MechanicInfo{
{"Avoiding Unkillable Objects",
"There are objects that the player cannot touch. These are different from normal enemies because they cannot be destroyed or moved."},
{"Race",
"The player must reach a place before the opponent does. Like \"Timed\" except the enemy as a \"timer\" can be slowed down by the player's actions, or there may be multiple enemies being raced against."},
}
// Get a random mechanic
func RandMechanic() *MechanicInfo {
i := rand.Intn(len(GameMechanics))
return GameMechanics[i]
}
var index = template.Must(template.ParseFiles("templates/index.html"))
func randMechHandler(w http.ResponseWriter, req *http.Request) {
mechanics := [3]*MechanicInfo{RandMechanic(), RandMechanic(), RandMechanic()}
if err := index.Execute(w, mechanics); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
func main() {
http.HandleFunc("/", randMechHandler)
if err := http.ListenAndServe(":80", nil); err != nil {
panic(err)
}
}