Asynchronous Testing With Stream Processing - asynchronous

I'm very new to Go, so I may be misunderstanding something foundational about Go's async/stream handling, but here goes...
I'm trying to write some tests using ginkgo on a function I wrote that processes streams.
The processing side reads in newline-delimited text from a File until it encounters a special delimiter line at which point it tries to parse the text as JSON. The code looks like this:
func ParseConfig(inStream *os.File) (Config, error){
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break;
}
header += line
}
// parse JSON here and return
}
My test looks something like this
Describe("ParseConfig()", func() {
It("should pass for a valid header", func(){
_, err := io.WriteString(stream, "{\"Key\": \"key\", \"File\": \"file\"}\n|||\n")
Expect(err).NotTo(HaveOccurred())
conf, err := parser.ParseConfig(stream)
Expect(err).NotTo(HaveOccurred())
Expect(conf.Key).To(Equal("key"))
})
})
Unfortunately, this yields a JSON parsing error, as it's trying to parse an empty string. I'm assuming that my problem is that I'm sending the string on the stream before I've told the ParseConfig() function to listen on that string for data? But I'm not entirely clear how I could refactor this to use proper go routines to first listen for data then send it.
Some of the potential solutions I saw were around the use of "channels" (with which I'm unfamiliar) but I was worried that this one need might not be worth a major refactor to introduce a whole new paradigm of concurrency.
Thanks!

Not sure if I understood correctly, but your ParseConfig should probably take an io.Reader instead of a *os.File. That way you can test it directly without worrying about concurrency.
file t_test.go:
package main
import (
"strings"
"testing"
"github.com/onsi/ginkgo"
"github.com/onsi/gomega"
)
var _ = ginkgo.Describe("ParseConfig()", func() {
ginkgo.It("should pass for a valid header", func() {
// really don't know what you were doing with your 'stream' variable
// This is a test, you should forge a test scenario and pass it to your config function
stream := strings.NewReader(`{"Key": "key", "File": "file"}` + "\n|||\n")
conf, err := ParseConfig(stream)
gomega.Expect(err).NotTo(gomega.HaveOccurred())
gomega.Expect(conf.Key).To(gomega.Equal("key"))
})
})
func TestParseConfig(t *testing.T) {
ginkgo.RunSpecs(t, "Parse Config")
}
file main.go
package main
import (
"bufio"
"encoding/json"
"io"
"log"
"os"
)
type Config struct {
Key string
File string
}
func ParseConfig(inStream io.Reader) (*Config, error) {
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break
}
header += line
}
c := &Config{}
// parse JSON here and return
if err := json.Unmarshal([]byte(header), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
f, err := os.Open("config.json")
if err != nil {
log.Fatal(err)
}
ParseConfig(f)
}

Related

Pull info from private site that requires login (Golang)

I am trying to pull my classes from my online timetable, however, it seems as if I cannot get past the login stage. My code is:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"strings"
"github.com/PuerkitoBio/goquery"
)
type App struct {
Client *http.Client
}
type Timetable struct {
Name string
}
const (
baseURL string = "https://myclasswebsite.com"
)
func (app *App) login() {
//login := loginInfo()
client := app.Client
loginURL := baseURL + "/portal2/#!/login"
data := url.Values{
"inputEmail": {"my_actual_username"},
"password": {"my_actual_password"},
}
response, err := client.PostForm(loginURL, data)
if err != nil {
log.Fatalln(err)
}
defer response.Body.Close()
_, err = ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalln(err)
}
}
func (app *App) getTimetable() []Timetable {
timetableURL := baseURL + "/portal/dashboard"
client := app.Client
response, err := client.Get(timetableURL)
if err != nil {
log.Fatalln("Error fetching response. ", err)
}
defer response.Body.Close()
document, err := goquery.NewDocumentFromReader(response.Body)
fmt.Println(document.Html())
if err != nil {
log.Fatal("Error loading HTTP response body. ", err)
}
var classes []Timetable
document.Find(".timetable table").Each(func(i int, s *goquery.Selection) {
className := strings.TrimSpace(s.Text())
class := Timetable{
Name: className,
}
classes = append(classes, class)
})
return classes
}
I changed the base URL and login info just for privacy reasons, however, the rest of the code is as-is.
My main. go file is:
package main
import (
"fmt"
"net/http"
"net/http/cookiejar"
)
func main() {
jar, _ := cookiejar.New(nil)
app := App{
Client: &http.Client{Jar: jar},
}
app.login()
classes := app.getTimetable()
fmt.Println("class array is", classes)
for index, class := range classes {
fmt.Printf("%d: %s\n", index+1, class.Name)
}
}
The final print returns an empty slice, and when I print the response.Html() to the console, I receive the login-page Html rather than the dashboard-page HTML.
I'm in no way expecting anyone to fix this for me but a second pair of eyes and maybe a clue in which direction I should go would be helpful. Thank you so much!
Since I'm unfamiliar with your class website, some ideas for progressing:
You're not checking the status code from your login call. You may be getting a non-200 status code.
After you've confirmed the status code, check the cookie jar to ensure that a cookie has been saved. This is probably on the unlikelier side, but it's worth checking.
Lastly, attempt the same sequence with curl with -v (if you haven't already). That will give you more insight as to what is happening with your call plan.
I ended up ignoring some cookies accidentally while testing. Whoops...

How can we update a record by a http post method in GoLang?

Problem description:
I am learning Golang to implement the REST API for a small project.
I was following this small example to get the some idea how to connect things.
However, it looks like there are some bugs in the sample example, that i could not get the expected response in postman after hitting the endpoints.
I have fixed it by adding the missing functions (HandleFunc functions) to make it work.
Problem Description:
However, I still have an issue with CreateEvent section.
The expectation is that after using POST method with a given sample Event (json format) like below, event list is updated.
{
"id": "23",
"title": "This is simple Go lang title for test!",
"Description":"In this course you will learn REST api implementation in Go lang"
}
But after reaching the "http://localhost:8080/events" endpoint in which I defined to return all the events (1 defined inside code, the other should be added by calling CreateEvent function) i get only one of the event (hard coded one inside code only) in response.
Here is the complete code.
I appreciate for any suggestions/comments.
package main
import (
"fmt"
"log"
"net/http"
"io/ioutil"
"encoding/json"
"github.com/gorilla/mux"
)
func homeLink(w http.ResponseWriter, r *http.Request) {
fmt.Println("test started!")
fmt.Fprintf(w, "Welcome home!")
}
func main() {
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/", homeLink)
/*i have added the next 3 lines, missing in the sample code*/
router.HandleFunc("/event", createEvent)
router.HandleFunc("/events/{id}", getOneEvent)
router.HandleFunc("/events", getAllEvents)
log.Fatal(http.ListenAndServe(":8080", router))
}
type event struct {
ID string `json:"ID"`
Title string `json:"Title"`
Description string `json:"Description"`
}
type allEvents []event
var events = allEvents{
{
ID: "1",
Title: "Introduction to Golang",
Description: "Come join us for a chance to learn how golang works and get to eventually try it out",
},
}
func createEvent(w http.ResponseWriter, r *http.Request) {
var newEvent event
reqBody, err := ioutil.ReadAll(r.Body)
if err != nil {
fmt.Fprintf(w, "Kindly enter data with the event title and description only in order to update")
}
fmt.Println("Create Event is called!")
json.Unmarshal(reqBody, &newEvent)
events = append(events, newEvent)
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(newEvent)
}
func getOneEvent(w http.ResponseWriter, r *http.Request) {
eventID := mux.Vars(r)["id"]
fmt.Println("get one event is called!")
fmt.Println(eventID)
for _, singleEvent := range events {
if singleEvent.ID == eventID {
json.NewEncoder(w).Encode(singleEvent)
}
}
}
func getAllEvents(w http.ResponseWriter, r *http.Request) {
fmt.Println("Get all events is called!")
json.NewEncoder(w).Encode(events)
}
Your code is working fine. I have tested it (just copied above code and ran in my local machine and tested with Postman).
Btw, i have added few recommendations for a better code below.
If there is not nil error, handle it and return.
reqBody, err := ioutil.ReadAll(r.Body)
if err != nil {
fmt.Fprintf(w, "Kindly enter data with the event title and description only in order to update")
return //add this return, otherwise continue the function with the error
}
Put this json error handling for createEvent Handler function
err = json.Unmarshal(reqBody, &newEvent)
if err != nil {
fmt.Fprintf(w, "json format invalid")
return
}
Add http methods to your endpoints.
router.HandleFunc("/", homeLink).Methods(http.MethodGet)
/*i have added the next 3 lines, missing in the sample code*/
router.HandleFunc("/event", createEvent).Methods(http.MethodPost)
router.HandleFunc("/events/{id}", getOneEvent).Methods(http.MethodGet)
router.HandleFunc("/events", getAllEvents).Methods(http.MethodGet)

Check if a URL is reachable using Golang

I want to create a simple script that checks if a certain hostname:port is running. I only want to get a bool response if that URL is live, but I'm not sure if there's a straightforward way of doing it.
If you only want see if a URL is reachable you could use net.DialTimeout. Like this:
timeout := 1 * time.Second
conn, err := net.DialTimeout("tcp","mysyte:myport", timeout)
if err != nil {
log.Println("Site unreachable, error: ", err)
}
If you want to check if a Web server answers on a certain URL, you can invoke an HTTP GET request using net/http.
You will get a timeout if the server doesn't response at all. You might also check the response status.
resp, err := http.Get("http://google.com/")
if err != nil {
print(err.Error())
} else {
print(string(resp.StatusCode) + resp.Status)
}
You can change the default timeout by initializing a http.Client.
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
resp, err := client.Get("http://google.com")
Bonus:
Go generally does not rely on exceptions and the built in libraries generally do not panic, but return an error as a second value.
See Why does Go not have exceptions?.
You can assume that something very bad happened if your call to a native function panics.
You can make a HEAD request:
package main
import "net/http"
func head(s string) bool {
r, e := http.Head(s)
return e == nil && r.StatusCode == 200
}
func main() {
b := head("https://stackoverflow.com")
println(b)
}
https://golang.org/pkg/net/http#Head
If you don't mind the port, use http.Get(web):
package main
import (
"fmt"
"net/http"
"os"
)
func main() {
web := os.Args[1]
fmt.Println(webIsReachable(web))
}
func webIsReachable(web string) bool {
response, errors := http.Get(web)
if errors != nil {
_, netErrors := http.Get("https://www.google.com")
if netErrors != nil {
fmt.Fprintf(os.Stderr, "no internet\n")
os.Exit(1)
}
return false
}
if response.StatusCode == 200 {
return true
}
return false
}

Go - Using closures and adding maps to slices - filesystem.walk

In the following code I am trying to add file information to a map, which gets added to a slice. The slice contains a map for every file on the filesystem. My understanding is this is similar to how you would build an array of dictionaries in Python.
package files
import (
"path/filepath"
"os"
"fmt"
)
func GetFiles() {
var numScanned int
var fileSlice = []map{}
numScanned = 0
var scan = func(path string, f os.FileInfo, err error) error {
numScanned ++
var fileDetails map[string]interface{}
fmt.Printf("%s with %d bytes\n", path,f.Size())
fileDetails["filename"] = f.Name()
fileDetails["filesize"] = f.Size()
fileSlice = append(fileSlice, fileDetails) //Error = cannot use fileDetails (type map[string]interface {}) as type string in append
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("%s with %d bytes\n", path,f.Size())
}
return nil
}
directories := [] string {"C:\\"}
numOfDirectories := len(directories)
for i := 0; i < numOfDirectories; i++ {
err := filepath.Walk(directories[i], scan)
if err != nil {
fmt.Printf(err.Error())
}
}
fmt.Printf("%d", numScanned)
}
Error:
c:\project\src>go run main.go
# files
C:\project\src\files\files.go:12: syntax error: unexpected ]
C:\project\src\files\files.go:14: non-declaration statement outside function body
C:\project\src\files\files.go:36: non-declaration statement outside function body
C:\project\src\files\files.go:37: non-declaration statement outside function body
C:\project\src\files\files.go:44: non-declaration statement outside function body
C:\project\src\files\files.go:46: non-declaration statement outside function body
C:\project\src\files\files.go:48: syntax error: unexpected }
I'm also trying to work out how closures work in Go, so I can add file details to a map which is added to a slice, which I can access after filesystem.Walk has finished.
Thanks
EDIT:
I updated the code but how do you make a slice of map?

Get gorilla/mux router current route name from middleware

Problem:
Unable to access mux.CurrentRoute(r).GetName() from middleware. (Although I had been able to access it from my middleware, I had to change the way my middleware works due to it's previous inability to access the request). So I've mucked something up and I'm not sure how to get back to a working state where I can access the route name.
Any help would be much appreciated!
Error:
runtime error: invalid memory address or nil pointer dereference
Code:
func main() {
var (
err error
r *mux.Router
devRouter *mux.Router
usersRouter *mux.Router
brandsRouter *mux.Router
)
defer db.Close()
defer store.Close()
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
usersRouter = r.PathPrefix("/api/v1/users").Subrouter()
brandsRouter = r.PathPrefix("/api/v1/brands").Subrouter()
// development endpoints
devRouter.HandleFunc("/db/seed", devDbSeed)
...
// users
usersRouter.HandleFunc("/create", usersCreateHandlerFunc).Methods("POST").Name("USERS_CREATE")
...
// brands
brandsRouter.HandleFunc("/create", brandsCreateHandlerFunc).Methods("POST").Name("BRANDS_CREATE")
...
// products
brandsRouter.HandleFunc("/{brand_id:[0-9]+}/products", brandsProductsListHandlerFunc).Methods("GET").Name("BRANDS_PRODUCTS_LIST")
...
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
http.Handle("/", use(r, mw...))
err = http.ListenAndServe(":9000", nil)
if err != nil {
logIt(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if true != authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
} else {
next.ServeHTTP(w, r)
}
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
/**
This is where it's failing!
*/
routeName := mux.CurrentRoute(r).GetName()
switch routeName {
case "USERS_CREATE":
// route-specific authorization
break
...
default:
break
}
return
}
Update (2015-01-04 # 4:49PM EST):
So after removing the middleware (or at least commenting out the section that's trying to read mux.CurrentRoute) I am able to retrieve the route name from the destination handlerfunc (ex: usersCreateHandlerFunc or brandsCreateHandlerFunc). This doesn't solve my problem (I'd still like to perform authentication/authorization in middleware as opposed to every handlerfunc), I have a hunch it's letting me know *mux.Router isn't available in my middleware until after the final .ServeHTTP call. (Or something along those lines...)
Update (2015-01-04 # 5:41PM EST):
Tried a different (albeit less-preferred) direction of using Negroni as the middleware component. Still getting nil-pointer error when I try to get mux.CurrentRoute.
Update (2015-01-04 # 6:17PM EST):
I am able to access the request (ex: r.URL) from the middleware func's, but still no luck on accessing the mux.Route (ex: mux.CurrentRoute(r)). After looking a bit more at the mux source, I think it's because the current mux context isn't getting set because the router hasn't executed the matcher yet (and therefore it doesn't know what route it's currently on until AFTER the middleware is complete). However, I'm still not sure how to either resolve this, or re-structure my code to handle this.
What about:
routeName := mux.CurrentRoute(r).GetName()
Where r is the *http.Request. Don't forget to import "github.com/gorilla/mux". Remember that in order to use this, you must give you route a name when you define it
From CurrentRoute godoc:
CurrentRoute returns the matched route for the current request, if any. This only works when called inside the handler of the matched route because the matched route is stored in the request context[...]
In your example, your chain of mwAuthenticate, mwAuthorize is attached to the route "/" without using gorilla mux. That means when the request passes your handlers, it has not passed gorilla mux router.
Try the following (your example stripped down):
package main
import (
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
)
var (
err error
r *mux.Router
devRouter *mux.Router
)
func devDbSeed(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "devDbSeed")
return
}
func main() {
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
// development endpoints
devRouter.Handle("/db/seed", use(http.HandlerFunc(devDbSeed), mw...)).Name("foo")
// Send all requests into the mux router
err = http.ListenAndServe(":9000", r)
if err != nil {
log.Fatal(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
})
}
func mwAuthenticate(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
next.ServeHTTP(w, r)
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
handlerName := "UNKNOWN"
if route := mux.CurrentRoute(r); route != nil {
routeName := route.GetName()
if routeName != "" {
handlerName = routeName
}
}
log.Println(handlerName)
switch handlerName {
case "USERS_CREATE":
// route-specific authorization
log.Println("USERS_CREATE")
break
default:
break
}
return
}
I had the same problem and I resolved in that way:
var match mux.RouteMatch
routeExists := s.Router.Match(r, &match)
if routeExists && match.Route.GetName(){
routeName := match.Route.GetName()
}
And when I defined the route I added .Name("route/:param") where route/:param is my route.

Resources