I hope you can help me out since this gave me quite a headache.
I'm creating a chain for the middleware which is executed afterwards. But it looks like it has become recurring. The variable next within the anonymous function points to itself.
type MiddlewareInterface interface {
// Run the middleware for the given request, and receive the next handler.
Run(http.ResponseWriter, *http.Request, http.Handler)
}
createChain(collection []MiddlewareInterface, handler http.Handler) http.Handler
next := handler
for _, middlew := range collection {
next = func(w http.ResponseWriter, res *http.Request) {
middlew.Run(w, res, next)
}
}
return next
}
I know it's kind of a noob question, but I sincerely do want to understand what causes this and how this can be resolved. Looking forward to your answers!
Seems this is a closure variable in loop problem. You are creating a function which is capturing next in each loop, but what this will mean is that all of the functions share the same variable next and they will all have the value that is left on the last loop. I think you can fix it by introducing a new temporary variable inside the loop scope:
func createChain(collection []MiddlewareInterface, handler http.Handler) http.Handler
next := handler
for _, middlew := range collection {
thisNext:= next
mw := middlew
next = func(w http.ResponseWriter, res *http.Request) {
mw.Run(w, res, thisNext)
}
}
return next
}
Possibly the placement of the new variable definition isn't quite right but the closure issue will definitely be the source of your problem. It's not normally how http middleware handlers work as they normally wrap each other rather than being chained.
Related
I have a very strange problem, and i'm either really blind, or this is some kind of a bug. I have the following http.Handler:
func ServeHTTP(w http.ResponseWriter, r *http.Request) {
err := r.ParseForm()
if err != nil {
log.Println("Error while parsing form data")
return
}
log.Println("Printing r.PostForm:")
for key, values := range r.PostForm { // range over map
for _, value := range values { // range over []string
log.Println(key, value)
}
}
b, _ := ioutil.ReadAll(r.Body)
s := string(b)
log.Println("Printing body: ",s)
}
Now, when sending a PUT-Request to the url binded to this handler with the following FORM-Data:
Name=someName
Version=1.0.0
PLanguage=java
GitRepo=someRepo
This is ALWAYS the output:
Printing r.PostForm:
Printing body: Name=someName&Version=1.0.0&PLanguage=java&GitRepo=someRepo
I've been trying to find the cause for like 2 hours already and i just have no idea what the heck is wrong here. There is no error parsing the Form-Data, but the r.PostForm map is always empty (i also tried r.Form, with same result). So for debugging i added the part where i print the body, just to make sure there actually is some data in there - and it is. I would really appreciate any help here. Thanks in advance!
You need to set the 'Content-Type' header.
If no header is set "application/octet-stream" is used according to RFC 2616.
Long story short that is a binary format so your body will not be parsed into the Form.
I'm new to Go and I'm learning how to work with goroutines.
I have a function that downloads images:
func imageDownloader(uri string, filename string) {
fmt.Println("starting download for ", uri)
outFile, err := os.Create(filename)
defer outFile.Close()
if err != nil {
os.Exit(1)
}
client := &http.Client{}
req, err := http.NewRequest("GET", uri, nil)
resp, err := client.Do(req)
defer resp.Body.Close()
if err != nil {
panic(err)
}
header := resp.ContentLength
bar := pb.New(int(header))
rd := bar.NewProxyReader(resp.Body)
// and copy from reader
io.Copy(outFile, rd)
}
When I call by itself as part of another function, it downloads images completely and there is no truncated data.
However, when I try to modify it to make it a goroutine, images are often truncated or zero length files.
func imageDownloader(uri string, filename string, wg *sync.WaitGroup) {
...
io.Copy(outFile, rd)
wg.Done()
}
func main() {
var wg sync.WaitGroup
wg.Add(1)
go imageDownloader(url, file, &wg)
wg.Wait()
}
Am I using WaitGroups incorrectly? What could cause this and how can I fix it?
Update:
Solved it. I had placed the wg.add() function outside of a loop. :(
While I'm not sure exactly what's causing your issue, here's two options for how to get it back into working order.
First, looking to the example of how to use waitgroups from the sync library, try calling defer wg.Done() at the beginning of your function to ensure that even if the goroutine ends unexpectedly, that the waitgroup is properly decremented.
Second, io.Copy returns an error that you're not checking. That's not great practice anyway, but in your particular case it's preventing you from seeing if there is indeed an error in the copying routine. Check it and deal with it appropriately. It also returns the number of bytes written, which might help you as well.
Your example doesn't have anything obviously wrong with its use of WaitGroups. As long as you are calling wg.Add() with the same number as the number of goroutines you launch, or incrementing it by 1 every time you start a new goroutine, that should be correct.
However you call os.Exit and panic for certain errors conditions in the goroutine, so if you have more than one of these running, a failure in any one of them will terminate all of them, regardless of the use of WaitGroups. If it's failing without a panic message, I would take a look at the os.Exit(1) line.
It would also, be good practice in go to use defer wg.Done() at the start of your function, so that even if an error occurs, the goroutine still decrements its counter. That way your main thread won't hang on completion if one of the goroutines returns an error.
One change I would make in your example is leverage defer when you are Done. I think this defer ws.Done() should be the first statement in your function.
I like WaitGroup's simplicity. However, I do not like that we need to pass the reference to the goroutine because that would mean that the concurrency logic would be mixed with your business logic.
So I came up with this generic function to solve this problem for me:
// Parallelize parallelizes the function calls
func Parallelize(functions ...func()) {
var waitGroup sync.WaitGroup
waitGroup.Add(len(functions))
defer waitGroup.Wait()
for _, function := range functions {
go func(copy func()) {
defer waitGroup.Done()
copy()
}(function)
}
}
So your example could be solved this way:
func imageDownloader(uri string, filename string) {
...
io.Copy(outFile, rd)
}
func main() {
functions := []func(){}
list := make([]Object, 5)
for _, object := range list {
function := func(obj Object){
imageDownloader(object.uri, object.filename)
}(object)
functions = append(functions, function)
}
Parallelize(functions...)
fmt.Println("Done")
}
If you would like to use it, you can find it here https://github.com/shomali11/util
I have a very, very large array (not slice) of maps that I am then trying to encode. I really need to avoid making a copy of the array but I can't figure out how to do this.
So I far I have this:
func doSomething() {
var mygiantvar [5]map[string]Searcher
mygiantvar = Load()
Save(`file.gob.gz`, &mygiantvar)
}
func Save(filename string, variable *[5]map[string]Searcher) error {
// Open file for writing
fi, err := os.Create(filename)
if err !=nil {
return err
}
defer fi.Close()
// Attach gzip writer
fz := gzip.NewWriter(fi)
defer fz.Close()
// Push from the gob encoder
encoder := gob.NewEncoder(fz)
err = encoder.Encode(*variable)
if err !=nil {
return err
}
return nil
}
From my understanding that will pass a pointer of mygiantvar to Save, which saves the first copy. But then the entire array will surely be copied into encoder.Encode which will then copy it around many more functions, right?
This mygiantvar variable will be something like 10GB in size. So it must avoid being copied ever.
But then again perhaps only the actual array [5] part is copied but the maps inside of this are pointers inside an array, so the array of pointers to maps would be copied instead of the maps themselves? I have no idea about this - it's all very confusing.
Any ideas?
Note that Encoder.Encode will pass around an interface{}.
func (enc *Encoder) Encode(v interface{}) error {
That means a kind of a pointer to whatever you will be passing to it, as I described in "what is the meaning of interface{} in golang?"
(see also "Why can't I assign a *Struct to an *Interface?")
An interface value isn't the value of the concrete struct (as it has a variable size, this wouldn't be possible), but it's a kind of pointer (to be more precise a pointer to the struct and a pointer to the type)
That means it won't copy the full content of your map (or here of your array).
Since array is a value, you could slice it to avoid any copy during the call to Encode():
err = encoder.Encode(*variable[:])
See "Go Slices: usage and internals"
This is also the syntax to create a slice given an array:
x := [3]string{"Лайка", "Белка", "Стрелка"}
s := x[:] // a slice referencing the storage of x
If that doesn't work, you can keep *variable (here an array: [5]map[string]Searcher), as map types are reference types, like pointers or slices: the copy won't be huge.
See "Go maps in action".
While the array will be copied when passed to interface{}, the map content won't be copied.
See this play.golang.org example:
package main
import "fmt"
func main() {
var a [1]map[string]int
a[0] = make(map[string]int)
a[0]["test"] = 0
modify(a)
fmt.Println(a)
}
func modify(arr interface{}) {
a := arr.([1]map[string]int)
a[0]["test"] = -1
}
Output:
[map[test:-1]]
Context
I'm making a web app that serves dynamically generated pdfs. These contain content from the internet, so every time it serves a pdf, it downloads a number of files to a new temporary folder.
The Problem
I end up with a large number of folders after I load the page once, so it seems that, for some reason, the handler is being called multiple times, which is an issue because I'm downloading multiple times more than I need to of not insubstantial files. I'd like to check at what stage of the process multiple requests are occurring.
The Question
Is there a way of working out how many times a function has been called, quite possibly using closures? (I haven't quite got closures into my mental model for programming yet; I don't completely understand them/how they're used).
This would preferably be something involving an int in the language rather than printing something at every stage and counting by hand - I'm looking for a more scalable solution than that (for later situations as well as this one).
Thanks!
Here are two ways you can count function calls, and one for method calls. There are plenty of other ways too, but just to get you started:
Using closure: (not what I would recommended)
package main
import(
"fmt"
"sync/atomic"
)
var Foo = func() (func() uint64) {
var called uint64
return func() uint64 {
atomic.AddUint64(&called, 1)
fmt.Println("Foo!")
return called
}
}()
func main() {
Foo()
c := Foo()
fmt.Printf("Foo() is called %d times\n", c)
}
Playground: http://play.golang.org/p/euKbamdI7h
Using global counter:
package main
import (
"fmt"
"sync/atomic"
)
var called uint64
func Foo() {
atomic.AddUint64(&called, 1)
fmt.Println("Foo!");
}
func main() {
Foo()
Foo()
fmt.Printf("Foo() is called %d times\n", called)
}
Playground: http://play.golang.org/p/3Ib29VCnoF
Counting method calls:
package main
import (
"fmt"
"sync/atomic"
)
type T struct {
Called uint64
}
func (t *T) Foo() {
atomic.AddUint64(&t.Called, 1)
fmt.Println("Foo!")
}
func main() {
var obj T
obj.Foo()
obj.Foo()
fmt.Printf("obj.Foo() is called %d times\n", obj.Called)
}
Playground: http://play.golang.org/p/59eOQdUQU1
Edit:
I just realized that the handler might not be in your own package. In such a case, you might want to write a wrapper:
var called uint64
func Foo() {
atomic.AddUint64(&called, 1)
importedPackage.Foo()
}
Edit 2:
Updated the examples to use atomic +1 operations.
Counting Calls
To answer the specific question you asked, here is one quick way to count handler executions:
func countCalls(h http.HandlerFunc) http.HandlerFunc {
var lock sync.Mutex
var count int
return func(w http.ResponseWriter, r *http.Request) {
lock.Lock()
count++
w.Header().Set("X-Call-Count", fmt.Sprintf("%d", count))
lock.Unlock()
h.ServeHTTP(w, r)
}
}
http.Handle("/foobar", countCalls(foobarHandler))
This will add a header that you can inspect with your favorite web developer tools; you could also just log it to standard output or something.
Logging Handlers
To expand upon the answers mentioned above, what you probably want to do to debug this and have in place for future use is to log details of each request.
package main
import (
"flag"
"log"
"net/http"
"os"
"github.com/gorilla/handlers"
)
var (
accessLogFile = flag.String("log", "/var/log/yourapp/access.log", "Access log file")
)
func main() {
accessLog, err := os.OpenFile(*accessLogFile, os.O_CREATE|os.O_WRITE|os.O_APPEND, 0644)
if err != nil {
log.Fatalf("Failed to open access log: %s", err)
}
wrap := func(f http.HandlerFunc) http.Handler {
return handlers.LoggingHandler(accessLog, http.HandlerFunc(foobarHandler))
}
http.Handle("/foobar", wrap(foobarHandler))
...
}
This uses LoggingHandler (or CombinedLoggingHandler) to write a standard Apache format log message that you can either inspect yourself or analyze with various tools.
An example of a log line would be
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
which tells you who made the request, when, what the method and URL was, how your server responded, and how long the response was. From this log, you should be able to see exactly what requests are being made, to determine not only how many times your handlers are being called, but exactly what is generating the requests and whether they're to another endpoint (like /favicon.ico).
I have a recursive function. The function will call itself with various different values depending on the data it gets, so the arity and depth of recursion is not known: each call may call itself zero or more times. The function may return any number of values.
I want to parallelise it by getting goroutines and channels involved. Each recursion of inner runs in its own goroutine, and sends back a value on the channel. The outer function deals with those values.
func outer(response []int) {
results := make([]int)
resultsChannel := make(chan int)
inner := func(...) {
resultsChannel <- «some result»;
// Recurse in a new goroutine.
for _, recursionArgument in «some calculated data» {
go inner(recursionArgument)
}
}
go inner(«initial values»);
for {
result := <- resultsChannel
results = append(results, result)
// HELP! How do I decide when to break?
}
return results
}
The problem comes with escaping the results channel loop. Because of the 'shape' of the recursion (unknown arity and depth) I can't say "finish after n events" and I can't send a sentinel value.
How do I detect when all my recursions have happened and return from outer? Is there a better way to approach this?
You can use a sync.WaitGroup to manage the collection of goroutines you spawn: call Add(1) before spawning each new goroutine, and Done when each goroutine completes. So something like this:
var wg sync.WaitGroup
inner := func(...) {
...
// Recurse in a new goroutine.
for _, recursionArgument := range «some calculated data» {
wg.Add(1)
go inner(recursionArgument)
}
...
wg.Done()
}
wg.Add(1)
go inner(«initial values»)
Now waiting on wg will tell you when all the goroutines have completed.
If you are reading the results from a channel, the obvious way to tell when there are no more results is by closing the channel. You can achieve this through another goroutine to do this for us:
go func() {
wg.Wait()
close(resultsChannel)
}()
You should now be able to simply range over resultsChannel to read all the results.