In the following code I am trying to add file information to a map, which gets added to a slice. The slice contains a map for every file on the filesystem. My understanding is this is similar to how you would build an array of dictionaries in Python.
package files
import (
"path/filepath"
"os"
"fmt"
)
func GetFiles() {
var numScanned int
var fileSlice = []map{}
numScanned = 0
var scan = func(path string, f os.FileInfo, err error) error {
numScanned ++
var fileDetails map[string]interface{}
fmt.Printf("%s with %d bytes\n", path,f.Size())
fileDetails["filename"] = f.Name()
fileDetails["filesize"] = f.Size()
fileSlice = append(fileSlice, fileDetails) //Error = cannot use fileDetails (type map[string]interface {}) as type string in append
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("%s with %d bytes\n", path,f.Size())
}
return nil
}
directories := [] string {"C:\\"}
numOfDirectories := len(directories)
for i := 0; i < numOfDirectories; i++ {
err := filepath.Walk(directories[i], scan)
if err != nil {
fmt.Printf(err.Error())
}
}
fmt.Printf("%d", numScanned)
}
Error:
c:\project\src>go run main.go
# files
C:\project\src\files\files.go:12: syntax error: unexpected ]
C:\project\src\files\files.go:14: non-declaration statement outside function body
C:\project\src\files\files.go:36: non-declaration statement outside function body
C:\project\src\files\files.go:37: non-declaration statement outside function body
C:\project\src\files\files.go:44: non-declaration statement outside function body
C:\project\src\files\files.go:46: non-declaration statement outside function body
C:\project\src\files\files.go:48: syntax error: unexpected }
I'm also trying to work out how closures work in Go, so I can add file details to a map which is added to a slice, which I can access after filesystem.Walk has finished.
Thanks
EDIT:
I updated the code but how do you make a slice of map?
Related
I am wondering how I can do something similar to this. I currently have multiple packages with that same structure and functions but they actually retrieve values from multiple APIs. I am also loading in a config that has an array with parameters to use one of those packages per array item.
I am wondering how I can create a variable that uses one of those packages based on the config value. Hopefully this is clear enough. Here is pseudo code that I have written up to explain. Thanks in advance
package main
import (
"errors"
"flag"
"os"
"project/lib"
"project/morelib"
"project/extralib"
"fmt"
"math"
"math/rand"
"time"
)
func stuff(info RunInfo) (err error) {
apiKey:= "stuff1" // actually in the config
apiSecret := "stuff2" // actually in the config
variable := lib.New(apiKey, apiSecret) //returns *Lib struct
//this is where I have to comment out the other libs and uncomment them as needed
// variable := morelib.New(apiKey, apiSecret)
// variable := extralib.New(apiKey, apiSecret)
//trying to switch between libraries like this or in a switch statement
if info.libname == "lib"{
variable = lib.New(apiKey, apiSecret) //.New("", "") returns *Lib struct
}else if info.libname == "morelib"{
variable = morelib.New(apiKey, apiSecret) //.New("", "") returns *MoreLib struct
}else if info.libname == "extralib"{
variable = extralib.New(apiKey, apiSecret) //.New("", "") returns *ExtraLib struct
}else{
err = errors.New("there was an error with the value.....")
return err
}
mystuffs, err := variable.GetBalance("USD")
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("mystuff value: %v", mystuffs.value)
return
}
type RunInfo struct{
libname string
//other stuff
}
func main() {
//load from config with array
config := config.Load()
for i:=0; i<compare; i++{
var runInfo RunInfo
runInfo.libname = config.libname
stuff(runInfo)
}
}
pseudo lib code:
func New(apiKey, apiSecret string) *Lib {
client := NewClient(apiKey, apiSecret)
return &Lib{client}
}
func NewClient(apiKey, apiSecret string) (c *client) {
return &client{apiKey, apiSecret, &http.Client{}, false}
}
type Lib struct {
client *client
}
type client struct {
apiKey string
apiSecret string
httpClient *http.Client
debug bool
}
func (b *Lib) GetBalance(currency string) (balance Balance, err error) {
payload, err := json.Marshal(BalanceParams{Currency: currency})
if err != nil {
return
}
r, err := b.client.do("POST", "GetBalance", string(payload), true)
if err != nil {
return
}
var response jsonResponse
if err = json.Unmarshal(r, &response); err != nil {
return
}
if err = handleErr(response); err != nil {
return
}
err = json.Unmarshal(response.Result, &balance)
return
}
Use and if statement as in the question, a switch statement or a map.
I assume that the type returned by the New function is the following interface:
type GetStuffer interface
GetStuff(string) (Stuff, error)
}
The switch statement is:
var variable GetStuffer
switch info.CompareVal {
case "lib":
variable = lib.New(string1, string2)
case "morelib":
variable = morelib.New(string1, string2)
case "extralib":
variable = extralib.New(string1, string2)
default:
return errors.New("there was an error with the value.....")
}
mystuffs, err := variable.GetMyStuff()
if err != nil {
fmt.Println(err)
return
}
For the map, initialize a package level variable with the map:
var m = map[string]func(string,string) GetStuffer {
"lib": lib.New,
"morelib": morelib.New,
"extralib": extralib.New,
}
and use it like this:
fn := m[info.CompareValue]
if m == nil {
return errors.New("there was an error with the value.....")
}
variable := fn(string1, string2)
mystuffs, err := variable.GetMyStuff()
if err != nil {
fmt.Println(err)
return
}
If the assumption above about the return type is not correct, then there are two options. The first and likely the simplest is to modify New functions to return type GetStuffer. If that's not possible, then write little adaptor functions:
var m = map[string]func(string,string) GetStuffer {
"lib":func(s1, s2 string) GetStuffer { return lib.New(s1, s2) }
"morelib":func(s1, s2 string) GetStuffer { return morelib.New(s1, s2) }
"extralib":func(s1, s2 string) GetStuffer { return extralib.New(s1, s2) }
}
Why don't you define an interface that's only one function? In your example would be
type Stuffer interface {
GetMyStuff(string) (Stuff, error)
}
Then you declare your variable as type Stuffer.
I'm writing minecraft server in Go, when server is being stressed by 2000+ connections I get this crash:
fatal error: concurrent map read and map write/root/work/src/github.com/user/imoobler/limbo.go:78 +0x351
created by main.main /root/work/src/github.com/user/imoobler/limbo.go:33 +0x368
My code:
package main
import (
"log"
"net"
"bufio"
"time"
"math/rand"
"fmt"
)
var (
connCounter = 0
)
func main() {
InitConfig()
InitPackets()
port := int(config["port"].(float64))
ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
if err != nil {
log.Fatal(err)
}
log.Println("Server launched on port", port)
go KeepAlive()
for {
conn, err := ln.Accept()
if err != nil {
log.Print(err)
} else {
connCounter+=1
go HandleConnection(conn, connCounter)
}
}
}
func KeepAlive() {
r := rand.New(rand.NewSource(15768735131534))
keepalive := &PacketPlayKeepAlive{
id: 0,
}
for {
for _, player := range players {
if player.state == PLAY {
id := int(r.Uint32())
keepalive.id = id
player.keepalive = id
player.WritePacket(keepalive)
}
}
time.Sleep(20000000000)
}
}
func HandleConnection(conn net.Conn, id int) {
log.Printf("%s connected.", conn.RemoteAddr().String())
player := &Player {
id: id,
conn: conn,
state: HANDSHAKING,
protocol: V1_10,
io: &ConnReadWrite{
rdr: bufio.NewReader(conn),
wtr: bufio.NewWriter(conn),
},
inaddr: InAddr{
"",
0,
},
name: "",
uuid: "d979912c-bb24-4f23-a6ac-c32985a1e5d3",
keepalive: 0,
}
for {
packet, err := player.ReadPacket()
if err != nil {
break
}
CallEvent("packetReceived", packet)
}
player.unregister()
conn.Close()
log.Printf("%s disconnected.", conn.RemoteAddr().String())
}
For now server is only "limbo".
Generally speaking (without having access to the code where the error occurs) you have a few options. Here are two of them:
sync.RWMutex
Control access to the map with sync.RWMutex{}. Use this option if you have single reads and writes, not loops over the map. See RWMutex
Here a sample with access control to someMap via someMapMutex:
var (
someMap = map[string]string{}
someMapMutex = sync.RWMutex{}
)
go func() {
someMapMutex.Lock()
someMap["key"] = "value"
someMapMutex.Unlock()
}()
someMapMutex.RLock()
v, ok := someMap["key"]
someMapMutex.RUnlock()
if !ok {
fmt.Println("key missing")
return
}
fmt.Println(v)
syncmap.Map
Use a syncmap.Map{} instead of a normal map. This map is already taking care of race issues but may be slower depending on your usage. syncmap.Map{}s main advantage lies with for loops. See syncmap
var (
someMap = syncmap.Map{}
)
go func() {
someMap.Store("key", "value")
}()
v, ok := someMap.Load("key")
if !ok {
fmt.Println("key missing")
return
}
fmt.Println(v)
// with syncmap, looping over all keys is simple without locking the whole map for the entire loop
someMap.Range(func(key, value interface{}) bool {
// cast value to correct format
val, ok := value.(string)
if !ok {
// this will break iteration
return false
}
// do something with key/value
fmt.Println(key, val)
// this will continue iterating
return true
})
General Advice
You should test your server with -race option and then eliminate all the race conditions it throws. That way you can easier eliminate such errors before they occur.
go run -race server.go
See golang race detector
I'd like to write a function that takes a SSH session and runs a command, providing an io.Reader that reads its output as it is recieved.
package main
import (
"golang.org/x/crypto/ssh"
"io"
)
func StreamOutput(session *ssh.Session, command string) (output io.Reader, err error) {
if session != nil {
defer session.Close()
}
if err != nil {
return output, err
}
// connect to both outputs
outReader, err := session.StdoutPipe()
if err != nil {
return output, err
}
errReader, err := session.StderrPipe()
if err != nil {
return output, err
}
output = io.MultiReader(outReader, errReader)
err = session.Start(command)
// return the readers that are to be read from
return output, err
When I test this code using an ioutil.ReadAll on output, however, it only prints the appropriate output intermittently, even after calling session.Wait(). Why might this be? Is there a better way to return the functions output (channels, perhaps?)? Thanks!
The problem was that I had the following code in my function:
if session != nil {
defer session.Close()
}
That way, the output would only print if the command completed and sent back its stdout before the function returned (which was infrequent). I fixed my code simply by removing these lines.
Problem:
Unable to access mux.CurrentRoute(r).GetName() from middleware. (Although I had been able to access it from my middleware, I had to change the way my middleware works due to it's previous inability to access the request). So I've mucked something up and I'm not sure how to get back to a working state where I can access the route name.
Any help would be much appreciated!
Error:
runtime error: invalid memory address or nil pointer dereference
Code:
func main() {
var (
err error
r *mux.Router
devRouter *mux.Router
usersRouter *mux.Router
brandsRouter *mux.Router
)
defer db.Close()
defer store.Close()
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
usersRouter = r.PathPrefix("/api/v1/users").Subrouter()
brandsRouter = r.PathPrefix("/api/v1/brands").Subrouter()
// development endpoints
devRouter.HandleFunc("/db/seed", devDbSeed)
...
// users
usersRouter.HandleFunc("/create", usersCreateHandlerFunc).Methods("POST").Name("USERS_CREATE")
...
// brands
brandsRouter.HandleFunc("/create", brandsCreateHandlerFunc).Methods("POST").Name("BRANDS_CREATE")
...
// products
brandsRouter.HandleFunc("/{brand_id:[0-9]+}/products", brandsProductsListHandlerFunc).Methods("GET").Name("BRANDS_PRODUCTS_LIST")
...
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
http.Handle("/", use(r, mw...))
err = http.ListenAndServe(":9000", nil)
if err != nil {
logIt(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if true != authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
} else {
next.ServeHTTP(w, r)
}
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
/**
This is where it's failing!
*/
routeName := mux.CurrentRoute(r).GetName()
switch routeName {
case "USERS_CREATE":
// route-specific authorization
break
...
default:
break
}
return
}
Update (2015-01-04 # 4:49PM EST):
So after removing the middleware (or at least commenting out the section that's trying to read mux.CurrentRoute) I am able to retrieve the route name from the destination handlerfunc (ex: usersCreateHandlerFunc or brandsCreateHandlerFunc). This doesn't solve my problem (I'd still like to perform authentication/authorization in middleware as opposed to every handlerfunc), I have a hunch it's letting me know *mux.Router isn't available in my middleware until after the final .ServeHTTP call. (Or something along those lines...)
Update (2015-01-04 # 5:41PM EST):
Tried a different (albeit less-preferred) direction of using Negroni as the middleware component. Still getting nil-pointer error when I try to get mux.CurrentRoute.
Update (2015-01-04 # 6:17PM EST):
I am able to access the request (ex: r.URL) from the middleware func's, but still no luck on accessing the mux.Route (ex: mux.CurrentRoute(r)). After looking a bit more at the mux source, I think it's because the current mux context isn't getting set because the router hasn't executed the matcher yet (and therefore it doesn't know what route it's currently on until AFTER the middleware is complete). However, I'm still not sure how to either resolve this, or re-structure my code to handle this.
What about:
routeName := mux.CurrentRoute(r).GetName()
Where r is the *http.Request. Don't forget to import "github.com/gorilla/mux". Remember that in order to use this, you must give you route a name when you define it
From CurrentRoute godoc:
CurrentRoute returns the matched route for the current request, if any. This only works when called inside the handler of the matched route because the matched route is stored in the request context[...]
In your example, your chain of mwAuthenticate, mwAuthorize is attached to the route "/" without using gorilla mux. That means when the request passes your handlers, it has not passed gorilla mux router.
Try the following (your example stripped down):
package main
import (
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
)
var (
err error
r *mux.Router
devRouter *mux.Router
)
func devDbSeed(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "devDbSeed")
return
}
func main() {
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
// development endpoints
devRouter.Handle("/db/seed", use(http.HandlerFunc(devDbSeed), mw...)).Name("foo")
// Send all requests into the mux router
err = http.ListenAndServe(":9000", r)
if err != nil {
log.Fatal(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
})
}
func mwAuthenticate(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
next.ServeHTTP(w, r)
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
handlerName := "UNKNOWN"
if route := mux.CurrentRoute(r); route != nil {
routeName := route.GetName()
if routeName != "" {
handlerName = routeName
}
}
log.Println(handlerName)
switch handlerName {
case "USERS_CREATE":
// route-specific authorization
log.Println("USERS_CREATE")
break
default:
break
}
return
}
I had the same problem and I resolved in that way:
var match mux.RouteMatch
routeExists := s.Router.Match(r, &match)
if routeExists && match.Route.GetName(){
routeName := match.Route.GetName()
}
And when I defined the route I added .Name("route/:param") where route/:param is my route.
I'm very new to Go, so I may be misunderstanding something foundational about Go's async/stream handling, but here goes...
I'm trying to write some tests using ginkgo on a function I wrote that processes streams.
The processing side reads in newline-delimited text from a File until it encounters a special delimiter line at which point it tries to parse the text as JSON. The code looks like this:
func ParseConfig(inStream *os.File) (Config, error){
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break;
}
header += line
}
// parse JSON here and return
}
My test looks something like this
Describe("ParseConfig()", func() {
It("should pass for a valid header", func(){
_, err := io.WriteString(stream, "{\"Key\": \"key\", \"File\": \"file\"}\n|||\n")
Expect(err).NotTo(HaveOccurred())
conf, err := parser.ParseConfig(stream)
Expect(err).NotTo(HaveOccurred())
Expect(conf.Key).To(Equal("key"))
})
})
Unfortunately, this yields a JSON parsing error, as it's trying to parse an empty string. I'm assuming that my problem is that I'm sending the string on the stream before I've told the ParseConfig() function to listen on that string for data? But I'm not entirely clear how I could refactor this to use proper go routines to first listen for data then send it.
Some of the potential solutions I saw were around the use of "channels" (with which I'm unfamiliar) but I was worried that this one need might not be worth a major refactor to introduce a whole new paradigm of concurrency.
Thanks!
Not sure if I understood correctly, but your ParseConfig should probably take an io.Reader instead of a *os.File. That way you can test it directly without worrying about concurrency.
file t_test.go:
package main
import (
"strings"
"testing"
"github.com/onsi/ginkgo"
"github.com/onsi/gomega"
)
var _ = ginkgo.Describe("ParseConfig()", func() {
ginkgo.It("should pass for a valid header", func() {
// really don't know what you were doing with your 'stream' variable
// This is a test, you should forge a test scenario and pass it to your config function
stream := strings.NewReader(`{"Key": "key", "File": "file"}` + "\n|||\n")
conf, err := ParseConfig(stream)
gomega.Expect(err).NotTo(gomega.HaveOccurred())
gomega.Expect(conf.Key).To(gomega.Equal("key"))
})
})
func TestParseConfig(t *testing.T) {
ginkgo.RunSpecs(t, "Parse Config")
}
file main.go
package main
import (
"bufio"
"encoding/json"
"io"
"log"
"os"
)
type Config struct {
Key string
File string
}
func ParseConfig(inStream io.Reader) (*Config, error) {
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break
}
header += line
}
c := &Config{}
// parse JSON here and return
if err := json.Unmarshal([]byte(header), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
f, err := os.Open("config.json")
if err != nil {
log.Fatal(err)
}
ParseConfig(f)
}