Pull info from private site that requires login (Golang) - http

I am trying to pull my classes from my online timetable, however, it seems as if I cannot get past the login stage. My code is:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
"strings"
"github.com/PuerkitoBio/goquery"
)
type App struct {
Client *http.Client
}
type Timetable struct {
Name string
}
const (
baseURL string = "https://myclasswebsite.com"
)
func (app *App) login() {
//login := loginInfo()
client := app.Client
loginURL := baseURL + "/portal2/#!/login"
data := url.Values{
"inputEmail": {"my_actual_username"},
"password": {"my_actual_password"},
}
response, err := client.PostForm(loginURL, data)
if err != nil {
log.Fatalln(err)
}
defer response.Body.Close()
_, err = ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalln(err)
}
}
func (app *App) getTimetable() []Timetable {
timetableURL := baseURL + "/portal/dashboard"
client := app.Client
response, err := client.Get(timetableURL)
if err != nil {
log.Fatalln("Error fetching response. ", err)
}
defer response.Body.Close()
document, err := goquery.NewDocumentFromReader(response.Body)
fmt.Println(document.Html())
if err != nil {
log.Fatal("Error loading HTTP response body. ", err)
}
var classes []Timetable
document.Find(".timetable table").Each(func(i int, s *goquery.Selection) {
className := strings.TrimSpace(s.Text())
class := Timetable{
Name: className,
}
classes = append(classes, class)
})
return classes
}
I changed the base URL and login info just for privacy reasons, however, the rest of the code is as-is.
My main. go file is:
package main
import (
"fmt"
"net/http"
"net/http/cookiejar"
)
func main() {
jar, _ := cookiejar.New(nil)
app := App{
Client: &http.Client{Jar: jar},
}
app.login()
classes := app.getTimetable()
fmt.Println("class array is", classes)
for index, class := range classes {
fmt.Printf("%d: %s\n", index+1, class.Name)
}
}
The final print returns an empty slice, and when I print the response.Html() to the console, I receive the login-page Html rather than the dashboard-page HTML.
I'm in no way expecting anyone to fix this for me but a second pair of eyes and maybe a clue in which direction I should go would be helpful. Thank you so much!

Since I'm unfamiliar with your class website, some ideas for progressing:
You're not checking the status code from your login call. You may be getting a non-200 status code.
After you've confirmed the status code, check the cookie jar to ensure that a cookie has been saved. This is probably on the unlikelier side, but it's worth checking.
Lastly, attempt the same sequence with curl with -v (if you haven't already). That will give you more insight as to what is happening with your call plan.

I ended up ignoring some cookies accidentally while testing. Whoops...

Related

How to verify Firebase ID token on App Engine in Go

The Firebase documentation, Verify ID Tokens, explains how to Verify ID tokens using the Firebase Admin SDK.
But first the Firebase Admin SDK must be setup. After installing the Go Admin SDK, with go get firebase.google.com/go it must be initialized (I assume this code is placed in the HTTP server main function):
app, err := firebase.NewApp(context.Background(), nil)
if err != nil {
log.Fatalf("error initializing app: %v\n", err)
}
Question #1: I have no context in my App Engine app because in Migrating from the App Engine Go SDK it states:
Use request.Context() or your preferred context instead of using
appengine.NewContext.
AFACT the request.Context() is available from within an HTTP Handler Function. See the HandleFunc example on golang.org. Using the request context would require initializing the Firebase SDK for every HTTP request!
Question #2: How can I get a reference to my Firebase app from within an HTTP handler?
func main() {
// ------> Initialize the Firebase Go SDK. <------ \\
app, err := firebase.NewApp(context.Background(), nil)
if err != nil {
log.Fatalf("error initializing app: %v\n", err)
}
// My usual boilerplate for App Engine.
http.HandleFunc("/api/foo", fooHandler)
http.HandleFunc("/api/bar", barHandler)
port := os.Getenv("PORT")
if port == "" {
port = "8080"
log.Printf("Defaulting to port %s", port)
}
log.Printf("Listening on port %s", port)
if err := http.ListenAndServe(":"+port, nil); err != nil {
log.Fatal(err)
}
}
There are some very useful looking functions in the firebase/firebase-admin-go repo, but without a context (for the HTTP request?) and no way to share app with my HTTP handlers, I can't work-out how to use them:
// ==================================================================
// https://firebase.google.com/docs/auth/admin/verify-id-tokens
// ==================================================================
func verifyIDToken(ctx context.Context, app *firebase.App, idToken string) *auth.Token {
// [START verify_id_token_golang]
client, err := app.Auth(ctx)
if err != nil {
log.Fatalf("error getting Auth client: %v\n", err)
}
token, err := client.VerifyIDToken(ctx, idToken)
if err != nil {
log.Fatalf("error verifying ID token: %v\n", err)
}
log.Printf("Verified ID token: %v\n", token)
// [END verify_id_token_golang]
return token
}
As treethought pointed out, context is of little importance for you. You can use context.Background().
The context you are passing to firebase API is not the context you are interested in. That's the context used to goroutines management.
What you need in fact is a context that is provided by the http server you are using. You use that server's context to get the data from the request, parse it, and pass it as idToken.
For example, if you use gin, this is usually achieved by adding middleware that has a contract:
return func(c *gin.Context) {
....
c.Next()
}
Inside it you use your firebase app/client to call verifyIDToken and then decide if you want to chain the request further to your handlers(c.Next()) or abort(c.Abort())
You need to encapsulate the state, in our case firebase client, inside that middleware so you create it as an object/struct with a method that returns a function with the signature specified above.
Here's an example.
I didn't need the flexibility provided by a separate unAuthorized function, and also I store the token inside the cookie, so in my case it became:
package middleware
import (
"bytes"
"context"
"encoding/json"
firebase "firebase.google.com/go/v4"
"firebase.google.com/go/v4/auth"
"github.com/gin-gonic/gin"
"io/ioutil"
"log"
"net/http"
"strings"
)
type firebaseAuthMiddleware struct {
client *auth.Client
}
func CreateFirebaseMiddleware() *firebaseAuthMiddleware {
app := initFirebaseAppDefault(context.Background())
authClient := createAuthClient(context.Background(), app)
return &firebaseAuthMiddleware{
client: authClient,
}
}
func (fam *firebaseAuthMiddleware) FirebaseAuthMiddleware() gin.HandlerFunc {
return func(c *gin.Context) {
if strings.HasSuffix(c.Request.URL.Path, "/query") {
idTokenCookie, err := c.Request.Cookie("FIREBASE_ID_TOKEN")
if idTokenCookie == nil || err != nil {
buildUnauthorizedResponse(c)
return
}
var objmap graphQLMessage
var bodyBytes []byte
if c.Request.Body != nil {
bodyBytes, _ = ioutil.ReadAll(c.Request.Body)
}
// Restore the io.ReadCloser to its original state
c.Request.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))
json.Unmarshal(bodyBytes, &objmap)
//stringBody := string(body)
token, err := fam.client.VerifyIDToken(context.Background(), idTokenCookie.Value)
if err != nil {
buildUnauthorizedResponse(c)
return
}
if objmap.Variables["userId"] != token.UID {
buildUnauthorizedResponse(c)
return
}
}
c.Next()
}
}
func buildUnauthorizedResponse(c *gin.Context) {
c.JSON(http.StatusUnauthorized, gin.H{
"status": http.StatusUnauthorized,
})
c.Abort()
}
type graphQLMessage struct {
Variables map[string]string
Query string
OperationName string
}
func initFirebaseAppDefault(ctx context.Context) *firebase.App {
app, err := firebase.NewApp(ctx, nil)
if err != nil {
log.Fatalf("error initializing app: %v\n", err)
}
return app
}
func createAuthClient(ctx context.Context, app *firebase.App) *auth.Client {
client, err := app.Auth(ctx)
if err != nil {
log.Fatalf("error getting Auth client: %v\n", err)
}
return client
}
And then register the middleware in main:
package main
import (
"context"
"errors"
"github.com/99designs/gqlgen/graphql/handler"
"github.com/99designs/gqlgen/graphql/playground"
mw "github.com/wnd-engineering/storeback/middleware"
"log"
"runtime/debug"
)
func main() {
r := gin.Default()
r.Use(mw.CreateFirebaseMiddleware().FirebaseAuthMiddleware())
r.POST("/query", graphqlHandler())
r.GET("/", playgroundHandler())
....
}
I think you should be able to just create a new context context.Background() for example.

Golang Struct as Payload for POST Request

New to golang. I'm trying to make a POST request to an auth endpoint to get back a token for authing further requests. Currently the error I'm getting is missing "credentials". I've written the same logic in Python so I know what I am trying to do is what the system is expecting.
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/cookiejar"
"os"
)
type Auth struct {
Method string `json:"credentials"`
Email string `json:"email"`
Password string `json:"password"`
Mfa string `json:"mfa_token"`
}
func main() {
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter Email: ")
e, _ := reader.ReadString('\n')
fmt.Print("Enter Password: ")
p, _ := reader.ReadString('\n')
fmt.Print("Enter 2FA Token: ")
authy, _ := reader.ReadString('\n')
auth := Auth{"manual", e, p, authy}
j, _ := json.Marshal(auth)
jar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: jar,
}
req, err := http.NewRequest("POST", "https://internaltool.com/v3/sessions", bytes.NewBuffer(j))
if err != nil {
log.Fatal(err)
}
req.Header.Add("Accept-Encoding", "gzip, deflate, br")
res, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)
s := string(body)
if res.StatusCode == 400 {
fmt.Println("Bad Credentials")
fmt.Println(s)
return
}
}
The question is - am I properly marshalling the AUTH struct into JSON and adding it appropriately to the POST request? As the API is not even seeing the credentials key in the JSON I think I must be doing something wrong. Anything helps.
Here's a minimum viable example of using json.Marshal to convert a Struct to a JSON object in the context of a POST request.
Go's standard libraries are fantastic, there is no need to pull in external dependencies to do such a mundane thing.
func TestPostRequest(t *testing.T) {
// Create a new instance of Person
person := Person{
Name: "Ryan Alex Martin",
Age: 27,
}
// Marshal it into JSON prior to requesting
personJSON, err := json.Marshal(person)
// Make request with marshalled JSON as the POST body
resp, err := http.Post("https://httpbin.org/anything", "application/json",
bytes.NewBuffer(personJSON))
if err != nil {
t.Error("Could not make POST request to httpbin")
}
// That's it!
// But for good measure, let's look at the response body.
body, err := ioutil.ReadAll(resp.Body)
var result PersonResponse
err = json.Unmarshal([]byte(body), &result)
if err != nil {
t.Error("Error unmarshaling data from request.")
}
if result.NestedPerson.Name != "Ryan Alex Martin" {
t.Error("Incorrect or nil name field returned from server: ", result.NestedPerson.Name)
}
fmt.Println("Response from server:", result.NestedPerson.Name)
fmt.Println("Response from server:", result.NestedPerson.Age)
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
// NestedPerson is the 'json' field of the response, what we originally sent to httpbin
type PersonResponse struct {
NestedPerson Person `json:"json"` // Nested Person{} in 'json' field
}
As http.Client is relatively a low-level abstraction, gorequest(https://github.com/parnurzeal/gorequest) as an alternative is strongly recommended.
headers, queries, and body can be posted in any type, which is a bit more like what we often do in Python.

Check if a URL is reachable using Golang

I want to create a simple script that checks if a certain hostname:port is running. I only want to get a bool response if that URL is live, but I'm not sure if there's a straightforward way of doing it.
If you only want see if a URL is reachable you could use net.DialTimeout. Like this:
timeout := 1 * time.Second
conn, err := net.DialTimeout("tcp","mysyte:myport", timeout)
if err != nil {
log.Println("Site unreachable, error: ", err)
}
If you want to check if a Web server answers on a certain URL, you can invoke an HTTP GET request using net/http.
You will get a timeout if the server doesn't response at all. You might also check the response status.
resp, err := http.Get("http://google.com/")
if err != nil {
print(err.Error())
} else {
print(string(resp.StatusCode) + resp.Status)
}
You can change the default timeout by initializing a http.Client.
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
resp, err := client.Get("http://google.com")
Bonus:
Go generally does not rely on exceptions and the built in libraries generally do not panic, but return an error as a second value.
See Why does Go not have exceptions?.
You can assume that something very bad happened if your call to a native function panics.
You can make a HEAD request:
package main
import "net/http"
func head(s string) bool {
r, e := http.Head(s)
return e == nil && r.StatusCode == 200
}
func main() {
b := head("https://stackoverflow.com")
println(b)
}
https://golang.org/pkg/net/http#Head
If you don't mind the port, use http.Get(web):
package main
import (
"fmt"
"net/http"
"os"
)
func main() {
web := os.Args[1]
fmt.Println(webIsReachable(web))
}
func webIsReachable(web string) bool {
response, errors := http.Get(web)
if errors != nil {
_, netErrors := http.Get("https://www.google.com")
if netErrors != nil {
fmt.Fprintf(os.Stderr, "no internet\n")
os.Exit(1)
}
return false
}
if response.StatusCode == 200 {
return true
}
return false
}

Get gorilla/mux router current route name from middleware

Problem:
Unable to access mux.CurrentRoute(r).GetName() from middleware. (Although I had been able to access it from my middleware, I had to change the way my middleware works due to it's previous inability to access the request). So I've mucked something up and I'm not sure how to get back to a working state where I can access the route name.
Any help would be much appreciated!
Error:
runtime error: invalid memory address or nil pointer dereference
Code:
func main() {
var (
err error
r *mux.Router
devRouter *mux.Router
usersRouter *mux.Router
brandsRouter *mux.Router
)
defer db.Close()
defer store.Close()
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
usersRouter = r.PathPrefix("/api/v1/users").Subrouter()
brandsRouter = r.PathPrefix("/api/v1/brands").Subrouter()
// development endpoints
devRouter.HandleFunc("/db/seed", devDbSeed)
...
// users
usersRouter.HandleFunc("/create", usersCreateHandlerFunc).Methods("POST").Name("USERS_CREATE")
...
// brands
brandsRouter.HandleFunc("/create", brandsCreateHandlerFunc).Methods("POST").Name("BRANDS_CREATE")
...
// products
brandsRouter.HandleFunc("/{brand_id:[0-9]+}/products", brandsProductsListHandlerFunc).Methods("GET").Name("BRANDS_PRODUCTS_LIST")
...
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
http.Handle("/", use(r, mw...))
err = http.ListenAndServe(":9000", nil)
if err != nil {
logIt(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if true != authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
} else {
next.ServeHTTP(w, r)
}
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
/**
This is where it's failing!
*/
routeName := mux.CurrentRoute(r).GetName()
switch routeName {
case "USERS_CREATE":
// route-specific authorization
break
...
default:
break
}
return
}
Update (2015-01-04 # 4:49PM EST):
So after removing the middleware (or at least commenting out the section that's trying to read mux.CurrentRoute) I am able to retrieve the route name from the destination handlerfunc (ex: usersCreateHandlerFunc or brandsCreateHandlerFunc). This doesn't solve my problem (I'd still like to perform authentication/authorization in middleware as opposed to every handlerfunc), I have a hunch it's letting me know *mux.Router isn't available in my middleware until after the final .ServeHTTP call. (Or something along those lines...)
Update (2015-01-04 # 5:41PM EST):
Tried a different (albeit less-preferred) direction of using Negroni as the middleware component. Still getting nil-pointer error when I try to get mux.CurrentRoute.
Update (2015-01-04 # 6:17PM EST):
I am able to access the request (ex: r.URL) from the middleware func's, but still no luck on accessing the mux.Route (ex: mux.CurrentRoute(r)). After looking a bit more at the mux source, I think it's because the current mux context isn't getting set because the router hasn't executed the matcher yet (and therefore it doesn't know what route it's currently on until AFTER the middleware is complete). However, I'm still not sure how to either resolve this, or re-structure my code to handle this.
What about:
routeName := mux.CurrentRoute(r).GetName()
Where r is the *http.Request. Don't forget to import "github.com/gorilla/mux". Remember that in order to use this, you must give you route a name when you define it
From CurrentRoute godoc:
CurrentRoute returns the matched route for the current request, if any. This only works when called inside the handler of the matched route because the matched route is stored in the request context[...]
In your example, your chain of mwAuthenticate, mwAuthorize is attached to the route "/" without using gorilla mux. That means when the request passes your handlers, it has not passed gorilla mux router.
Try the following (your example stripped down):
package main
import (
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
)
var (
err error
r *mux.Router
devRouter *mux.Router
)
func devDbSeed(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "devDbSeed")
return
}
func main() {
r = mux.NewRouter()
devRouter = r.PathPrefix("/api/v1/dev").Subrouter()
// mwAuthorize and mwAuthenticate basically work the same
mw := []func(http.Handler) http.Handler{mwAuthenticate, mwAuthorize}
// development endpoints
devRouter.Handle("/db/seed", use(http.HandlerFunc(devDbSeed), mw...)).Name("foo")
// Send all requests into the mux router
err = http.ListenAndServe(":9000", r)
if err != nil {
log.Fatal(err)
}
}
func use(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
// exec order: mw[0],mw[1],mw[N]...
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
func mwAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !authorize(r) {
w.WriteHeader(http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
})
}
func mwAuthenticate(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
next.ServeHTTP(w, r)
})
}
func authorize(r *http.Request) (isAuthorized bool) {
isAuthorized = false
handlerName := "UNKNOWN"
if route := mux.CurrentRoute(r); route != nil {
routeName := route.GetName()
if routeName != "" {
handlerName = routeName
}
}
log.Println(handlerName)
switch handlerName {
case "USERS_CREATE":
// route-specific authorization
log.Println("USERS_CREATE")
break
default:
break
}
return
}
I had the same problem and I resolved in that way:
var match mux.RouteMatch
routeExists := s.Router.Match(r, &match)
if routeExists && match.Route.GetName(){
routeName := match.Route.GetName()
}
And when I defined the route I added .Name("route/:param") where route/:param is my route.

Asynchronous Testing With Stream Processing

I'm very new to Go, so I may be misunderstanding something foundational about Go's async/stream handling, but here goes...
I'm trying to write some tests using ginkgo on a function I wrote that processes streams.
The processing side reads in newline-delimited text from a File until it encounters a special delimiter line at which point it tries to parse the text as JSON. The code looks like this:
func ParseConfig(inStream *os.File) (Config, error){
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break;
}
header += line
}
// parse JSON here and return
}
My test looks something like this
Describe("ParseConfig()", func() {
It("should pass for a valid header", func(){
_, err := io.WriteString(stream, "{\"Key\": \"key\", \"File\": \"file\"}\n|||\n")
Expect(err).NotTo(HaveOccurred())
conf, err := parser.ParseConfig(stream)
Expect(err).NotTo(HaveOccurred())
Expect(conf.Key).To(Equal("key"))
})
})
Unfortunately, this yields a JSON parsing error, as it's trying to parse an empty string. I'm assuming that my problem is that I'm sending the string on the stream before I've told the ParseConfig() function to listen on that string for data? But I'm not entirely clear how I could refactor this to use proper go routines to first listen for data then send it.
Some of the potential solutions I saw were around the use of "channels" (with which I'm unfamiliar) but I was worried that this one need might not be worth a major refactor to introduce a whole new paradigm of concurrency.
Thanks!
Not sure if I understood correctly, but your ParseConfig should probably take an io.Reader instead of a *os.File. That way you can test it directly without worrying about concurrency.
file t_test.go:
package main
import (
"strings"
"testing"
"github.com/onsi/ginkgo"
"github.com/onsi/gomega"
)
var _ = ginkgo.Describe("ParseConfig()", func() {
ginkgo.It("should pass for a valid header", func() {
// really don't know what you were doing with your 'stream' variable
// This is a test, you should forge a test scenario and pass it to your config function
stream := strings.NewReader(`{"Key": "key", "File": "file"}` + "\n|||\n")
conf, err := ParseConfig(stream)
gomega.Expect(err).NotTo(gomega.HaveOccurred())
gomega.Expect(conf.Key).To(gomega.Equal("key"))
})
})
func TestParseConfig(t *testing.T) {
ginkgo.RunSpecs(t, "Parse Config")
}
file main.go
package main
import (
"bufio"
"encoding/json"
"io"
"log"
"os"
)
type Config struct {
Key string
File string
}
func ParseConfig(inStream io.Reader) (*Config, error) {
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break
}
header += line
}
c := &Config{}
// parse JSON here and return
if err := json.Unmarshal([]byte(header), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
f, err := os.Open("config.json")
if err != nil {
log.Fatal(err)
}
ParseConfig(f)
}

Resources