I'm a beginner to golang.
Is there any way to limit golang's http.Get() bandwidth usage? I found this: http://godoc.org/code.google.com/p/mxk/go1/flowcontrol, but I'm not sure how to piece the two together. How would I get access to the http Reader?
Thirdparty packages have convenient wrappers. But if you interested in how things work under the hood - it's quite easy.
package main
import (
"io"
"net/http"
"os"
"time"
)
var datachunk int64 = 500 //Bytes
var timelapse time.Duration = 1 //per seconds
func main() {
responce, _ := http.Get("http://google.com")
for range time.Tick(timelapse * time.Second) {
_, err :=io.CopyN(os.Stdout, responce.Body, datachunk)
if err!=nil {break}
}
}
Nothing magic.
There is an updated version of the package on github
You use it by wrapping an io.Reader
Here is a complete example which will show the homepage of Google veeeery sloooowly.
This wrapping an interface to make new functionality is very good Go style, and you'll see a lot of it in your journey into Go.
package main
import (
"io"
"log"
"net/http"
"os"
"github.com/mxk/go-flowrate/flowrate"
)
func main() {
resp, err := http.Get("http://google.com")
if err != nil {
log.Fatalf("Get failed: %v", err)
}
defer resp.Body.Close()
// Limit to 10 bytes per second
wrappedIn := flowrate.NewReader(resp.Body, 10)
// Copy to stdout
_, err = io.Copy(os.Stdout, wrappedIn)
if err != nil {
log.Fatalf("Copy failed: %v", err)
}
}
Related
I am trying to pull my classes from my online timetable, however, it seems as if I cannot get past the login stage. My code is:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"strings"
"github.com/PuerkitoBio/goquery"
)
type App struct {
Client *http.Client
}
type Timetable struct {
Name string
}
const (
baseURL string = "https://myclasswebsite.com"
)
func (app *App) login() {
//login := loginInfo()
client := app.Client
loginURL := baseURL + "/portal2/#!/login"
data := url.Values{
"inputEmail": {"my_actual_username"},
"password": {"my_actual_password"},
}
response, err := client.PostForm(loginURL, data)
if err != nil {
log.Fatalln(err)
}
defer response.Body.Close()
_, err = ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalln(err)
}
}
func (app *App) getTimetable() []Timetable {
timetableURL := baseURL + "/portal/dashboard"
client := app.Client
response, err := client.Get(timetableURL)
if err != nil {
log.Fatalln("Error fetching response. ", err)
}
defer response.Body.Close()
document, err := goquery.NewDocumentFromReader(response.Body)
fmt.Println(document.Html())
if err != nil {
log.Fatal("Error loading HTTP response body. ", err)
}
var classes []Timetable
document.Find(".timetable table").Each(func(i int, s *goquery.Selection) {
className := strings.TrimSpace(s.Text())
class := Timetable{
Name: className,
}
classes = append(classes, class)
})
return classes
}
I changed the base URL and login info just for privacy reasons, however, the rest of the code is as-is.
My main. go file is:
package main
import (
"fmt"
"net/http"
"net/http/cookiejar"
)
func main() {
jar, _ := cookiejar.New(nil)
app := App{
Client: &http.Client{Jar: jar},
}
app.login()
classes := app.getTimetable()
fmt.Println("class array is", classes)
for index, class := range classes {
fmt.Printf("%d: %s\n", index+1, class.Name)
}
}
The final print returns an empty slice, and when I print the response.Html() to the console, I receive the login-page Html rather than the dashboard-page HTML.
I'm in no way expecting anyone to fix this for me but a second pair of eyes and maybe a clue in which direction I should go would be helpful. Thank you so much!
Since I'm unfamiliar with your class website, some ideas for progressing:
You're not checking the status code from your login call. You may be getting a non-200 status code.
After you've confirmed the status code, check the cookie jar to ensure that a cookie has been saved. This is probably on the unlikelier side, but it's worth checking.
Lastly, attempt the same sequence with curl with -v (if you haven't already). That will give you more insight as to what is happening with your call plan.
I ended up ignoring some cookies accidentally while testing. Whoops...
I'm trying to access the HackerNews API to practice Go.
Anytime I go to my localhost to try see an output from Firebase database (where the data is stored) I am met with a Google Accounts authentication form.
Any help on this would be appreciated. In my terminal I used curl to check if I was getting a response from the server. I got a 200 OK response with content.
I thought I might be missing a Firebase client library but I'm not sure if that is the issue right now.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
response, err := http.Get("https://hacker-news.firebaseio.com/v0/item/8863.json")
if err != nil {
fmt.Printf("The http request failed with the error %s\n", err)
} else {
data, _ := ioutil.ReadAll(response.Body)
fmt.Fprintf(w, string(data))
}
}
func main() {
fmt.Println("Starting the applicaiton")
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
Aren't you suppose to add .json to URL (via https://github.com/HackerNews/API)?
e.g https://hacker-news.firebaseio.com/v0/item/8863.json
I'm trying to send data (files or whatever) through HTTP from the client to a server and read them as stream in the server.
But I noticed the chunk size or buffer size when the request's body is read it is fixed to 32kb. I tried doing it with TCP before using HTTP and the buffer size was the expected assigned size.
The data received from the request is being written to a file
Questions:
Is it possible to increase the chunk / buffer size?
if it is possible, by having a bigger buffer size will it increase performance due to less write calls to to the file being created?
If it is not possible, should I worry about performance loss by doing more write calls to the file being created?
Would it be better to use TCP? I really need the headers and http response
Here is some code for illustration:
client.go:
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
)
func main() {
addr := "http://localhost:8080"
path := "path/to/file"
sendHTTP(addr, path)
}
func sendHTTP(addr, path string) {
f, err := os.Open(path)
if err != nil {
log.Fatal("Error opening file:", err)
}
client := &http.Client{}
req, err := http.NewRequest("POST", addr, f)
if err != nil {
f.Close()
log.Fatal("Error creating request:", err)
}
_, err = client.Do(req)
if err != nil {
f.Close()
log.Fatal("Error doing request:", err)
}
}
server.go:
package main
import (
"fmt"
"io"
"log"
"net/http"
)
func main() {
addr := ":8080"
http.HandleFunc("/", handler)
http.ListenAndServe(addr, nil)
}
func handler(_ http.ResponseWriter, r *http.Request) {
buf := make([]byte, 512*1024) // 512kb
for {
br, err := r.Body.Read(buf)
if err == io.EOF {
break
} else if err != nil {
log.Println("Error reading request:", err)
break
}
fmt.Println(br) // is always 32kb
}
}
The call r.Body.Read(buf) waits for data from the network and returns up to len(buf) bytes of the available data. The amount of available data at the time of the call depends on timing and buffer sizes on the client, server and network. It's not easy to control.
The data received from the request is being written to a file
To write the data to the file in the most efficient way, copy from the request body to the file using io.Copy. Here's an example where f is the *os.File you want to write:
_, err := io.Copy(f, r.Body)
if err != nil {
// handle error
}
At the time I am writing this answer, the io.Copy function calls f.ReadFrom(r.Body) to copy the request body to a file.
I'm using Gorilla mux for all my routing. Now my app is working fine, I want to find a way to log all my response codes to -for example- statds. I have found this package: https://godoc.org/github.com/gorilla/handlers#LoggingHandler
Which allows me to output all responses into apache format. Although this is nice, it's not 100% what I want. I just want to extract the response statusses and send them to statds. Now what's the best/easiest way to achieve this?
package main
import (
"log"
"net/http"
"os"
"github.com/gorilla/handlers"
"github.com/gorilla/mux"
"github.com/rogierlommers/mux-status-handler/articles"
"github.com/rogierlommers/mux-status-handler/users"
)
func main() {
r := mux.NewRouter()
r.HandleFunc("/products", articles.Handler)
r.HandleFunc("/users", users.Handler)
loggedRouter := handlers.LoggingHandler(os.Stdout, r)
log.Println("listening on 8080")
http.ListenAndServe(":8080", loggedRouter)
}
Above code gives me this:
So I'm looking for something similar, but instead of outputting the Apache access logs to stdout, I would like to be able to "do something" with the response code. I have also created a simple repo which contains my sample code. You can find it here.
I found this useful Blog post from Tim Andersson.
First he builds a new struct that satisfies the interface:
type loggingResponseWriter struct {
http.ResponseWriter
statusCode int
}
func NewLoggingResponseWriter(w http.ResponseWriter) *loggingResponseWriter {
return &loggingResponseWriter{w, http.StatusOK}
}
func (lrw *loggingResponseWriter) WriteHeader(code int) {
lrw.statusCode = code
lrw.ResponseWriter.WriteHeader(code)
}
Then he's using it as a wrapper (or middleware):
func wrapHandlerWithLogging(wrappedHandler http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
log.Printf("--> %s %s", req.Method, req.URL.Path)
lrw := NewLoggingResponseWriter(w)
wrappedHandler.ServeHTTP(lrw, req)
statusCode := lrw.statusCode
log.Printf("<-- %d %s", statusCode, http.StatusText(statusCode))
})
}
This is how it can be made with violetear, probably can give you a hint about how to deal with the status code within the handler:
package main
import (
"fmt"
"log"
"net/http"
"github.com/nbari/violetear"
)
func handleGET(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("I handle GET requests\n"))
// do anything here with the Status code
cw := w.(*violetear.ResponseWriter)
fmt.Printf("The status code is: %d\n", cw.Status())
}
func main() {
router := violetear.New()
router.HandleFunc("/", handleGET, "GET")
log.Fatal(http.ListenAndServe(":8080", router))
}
By using:
cw := w.(*violetear.ResponseWriter)
You can access the violetear.ResponseWriter which exposes the status code by using cw.Status()
You can write your own middleware, here's a very base example
package main
import (
"log"
"net/http"
"github.com/gorilla/mux"
"github.com/rogierlommers/mux-status-handler/articles"
"github.com/rogierlommers/mux-status-handler/users"
)
// middleWare ...
func middleWare(handler http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// right not all this does is log like
// "github.com/gorilla/handlers"
log.Printf("%s %s %s", r.RemoteAddr, r.Method, r.URL)
// However since this is middleware you can have it do other things
// Examples, auth users, write to file, redirects, handle panics, ect
// add code to log to statds, remove log.Printf if you want
handler.ServeHTTP(w, r)
})
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/products", articles.Handler)
r.HandleFunc("/users", users.Handler)
log.Println("listening on 8080")
http.ListenAndServe(":8080", middleWare(r))
}
I want to stop the http server remotely (which I can do), but I also want to receive a message indicating that it has been stopped after it has been stopped. That is causing me some problems. The only solution that I could find is as shown below, which I do not consider ideal.
Can anyone provide a better solution. The problem is that the final message sent to the client is not getting through unless I use the goroutine as shown at the end "go func() {".
Code is as follows:
//*************
func stopServer(ohtWriter http.ResponseWriter, phtRequest *http.Request) {// Stop The Server
//*************
var iBytesSent int
var oOsError os.Error
var sErmes string
println("Stopping Server")
iBytesSent,oOsError = ohtWriter.Write([]byte("Message from server - server now stopped."))
if oOsError != nil {
sErmes = ". Error = " +oOsError.String()
} else {
sErmes = ". No error on write"
}
println("stopServer: Bytes sent = " +strconv.Itoa(iBytesSent) +sErmes)
ohtFlusher, tCanFlush := ohtWriter.(http.Flusher)
if tCanFlush {
ohtFlusher.Flush()
}
go func() {
time.Sleep(3e9)
os.Exit(0)
}()
}
Yeah I think without support from the http package, a graceful shutdown is not really possible. This is maybe a little bit less cringe-worthy, but will still slam closed any other concurrent requests in flight at the time of this request. Maybe try filing a feature request on the Go issue tracker. Better yet, open up the http package, and add a graceful shutdown method, and submit it.
Edit: I guess if you control all of the http.Handlers in your app, you could keep a count of in-flight requests (using appropriate thread synchronization), and modify the code below to a) refuse new connections once "shutdown" is called and b) wait for all in-flight requests to complete before shutting down...
package main
import (
"http"
"os"
"io"
"log"
"strconv"
)
func main() {
http.HandleFunc("/", ServeHTTP)
http.ListenAndServe(":8081", nil)
}
const responseString = "Shutting down\n"
func ServeHTTP(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.Header().Set("Content-Length", strconv.Itoa(len(responseString)))
io.WriteString(w, responseString)
f, canFlush := w.(http.Flusher)
if canFlush {
f.Flush()
}
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
log.Fatalf("error while shutting down: %v", err)
}
conn.Close()
log.Println("Shutting down")
os.Exit(0)
}
Haven't tried it yet, but using http.ServerConn directly might work.
Here's a simple way that's good enough for local development.
http://www.sergiotapia.me/how-to-stop-your-go-http-server/
package main
import (
"net/http"
"os"
"github.com/bmizerany/pat"
)
var mux = pat.New()
func main() {
mux.Get("/kill", http.HandlerFunc(kill))
http.Handle("/", mux)
http.ListenAndServe(":8080", nil)
}
func kill(w http.ResponseWriter, r *http.Request) {
os.Exit(0)
}